Main Page

Welcome to the PaskvilWiki!

The future shape of this wiki is unknown for now, let's hope it'll be something good...

This is partially a private wiki - viewing is allowed to all, but editing is allowed only to registered users.

This is more of a prevention than denial - I don't have time to monitor and moderate edits; - you're more than welcome to request registration, all contributors are very much welcome!

Just write me at josef.at.paskvil.dot.com.

Short Notes - Tips and Tricks

4store

• setup and installation
• running 4store
• data import
• data import and querying from Python
• Turtle file printer (in Python)

Apache

• Note on serving of local files
• Self-Signed SSL Certificate
• Enable SSL/HTTPS in Apache
• .htaccess and mod_rewrite Tricks

AWS

• Debugging (lack of) connection to instance
• Authorization header being removed by ElasticBeanstalk
• Notes on IAM
• AWS Lambda with Python - Notes
• boto3 Snippets
• AWS Chalice
• Building a Python Layer

Bash

• Extensions and File Name
• File Test Operators
• Commands and Shortcuts

$ sudo !!                         - run last command as root
$ !!:gs/foo/bar                   - run last command, replacing 'foo' with 'bar'
$ ^foo^bar                        - run last command, replacing 'foo' with 'bar'
$ cp filename{,.bak}              - create a .bak copy of file
<Ctrl-x> <Ctrl-e>                 - open $EDITOR to edit current command line; upon exit, content will be run
$ <space>command                  - run the command without saving it in history
$ > file.txt                      - empty a file
<Alt+.>                           - put last argument on cursor place; repeat to cycle through
$ (cd /tmp && ls)                 - jump to /tmp, execute ls, and jump back here
$ pushd /tmp .... $ popd          - goes to /tmp, remembering current location; at later time, popd will bring you back
$ \command                        - run command without using aliases
$ rm !(pattern|pattern)           - delete files that do not match given patterns
$ find . -mtime +5 -exec rm {} \; - delete all files in this folder that are older than 5 days
<Ctrl-u> ... <Ctrl-y>             - Ctrl+u will cut current command, then you can do whatever you need, and Ctrl+y will paste it back
$ date -d @1234567890             - convert unix timestamp to human-readable
$ jul () { date -d "$1-01-01 +$2 days -1 day" "+%Y%m%d"; }   - `jul $year $daynum` prints $daynum's day of $year, with Jan 1st being day 1

• Informations

$ mount | column -t               - pretty-print mount table
$ cat /etc/issue                  - display distro name

• Small Tricks

$ echo $(printf '%q' "A\"B")      - prints escaped string (in this case, > A\"B <)
$ mount -t tmpfs tmpfs /mnt -o size=1024m      - mount temporary RAM partition (fast access, non-persistent!)
$ disown -a && exit               - exit the shell while keeping all running processes alive
$ ssh -t remote_host screen -r    - directly connect to a remote screen process
$ for I in $(mysql -e 'show databases' -s --skip-column-names); do mysqldump $I | gzip > "$I.sql.gz"; done
                                  - backup all databases into individual files
$ fuser -k filename               - kill process that is locking given file
$ readom dev=/dev/scd0 f=/path/to/image.iso    - create CD/DVD ISO image
$ diff -rq folder-1 folder-2      - compare 2 folders (files missing or differing)
$ mkdir -p a/deep/dir/tree        - create nested directories fast
$ mount file.iso /mnt/iso -o loop - mount ISO file as drive /mnt/iso (dir has to preexist)
$ sudo touch /forcefsck           - force a file system check on the next reboot
$ ls | while read f; do process_file "$f"; done    - iterate through files even if they contain space in name
$ while read f; do process_file "$f"; done < <(ls) - iterates through files, but does not create sub-process (outside scope is visible)

• lsof

$ lsof -nPi                       - list network connections
$ netstat -tlnp                   - print all listening ports with PID's (run as root to see all process names)
$ lsof -c gnome-terminal          - list files opened by given command
$ lsof +D [folder]                - discover all open files in the [folder]
$ lsof -p [pid]                   - list files opened by given PID
$ lsof -i :8080                   - list programs that have port 8080 open ("who to kill" if you get "port already in use")
$ if [ `lsof -- file | wc -l` -ne 0 ] ... - tests if file is open by some process

C/C++

• How to clean up after child thread
• Non-blocking IO using sockets
• Shared Libraries on Linux+GCC
• Singleton
• std::move Semantics
• Converting between std::string and std::wstring in C++11

cURL (in terminal)

Use -s option to make curl silent.

• POST to server using cURL:

curl --data "key1=value1&key2=value2" http://www.example.com/post.php

• Upload to FTP server (with credentials), having connection timeout 60 seconds, and 120 seconds maximum operation time:

curl --connect-timeout 60 -m 120 -T file.ext "ftp://12.34.56.78/folder" --user user:pass

• Download from FTP server (with credentials):

curl --user user:pass "ftp://12.34.56.78/folder/file.ext" -o ~/Downloads/file.ext

ElasticSearch

• Installation
• What You Get
• Indexing, Routing, etc.
• Deleting
• Getting and Multi-Getting
• Updating
• Search and Query DSL

ffmpeg

• process all files, may include spaces in file name:

ls *.avi | while read f; do ffmpeg -i "$f" ...params... "$f.mkv"; done

• if you're having problems with input format, passing video through mpeg typically helps:

ffmpeg -i file.mkv -qscale 2 -f mpeg - | ffmpeg -i - ...params... file.avi

• downmix 5.1 dolby to normal stereo:

ffmpeg -i file.mkv ...params... -acodec libmp3lame -ar 44100 -ac 2 file.avi

• add an overlay image to the video:

ffmpeg -i input.mp4 -i image.png -filter_complex "[0:v][1:v] overlay=x:y:enable='between(t,0,20)'" -pix_fmt yuv420p -c:a copy output.mp4

This will add image.png as an overlay at position x, y over the imput.mp4 video, for the first 20 seconds.

• fade in and out, first and last 25 frames of a 1000 frame video

ffmpeg -i input.mp4 -vf "fade=in:0:25,fade=out:975:25" -acodec copy out.mp4

• convert album contained in APE or similar file with CUE into separate MP3 tracks

ffmpeg -i cd.ape -ab 256k cd.mp3
mp3splt -a -c cd.cue cd.mp3

more on ffmpeg

• Rip DVD into FLV (or anything else)
• Video File with Still Image and Audio
• Concatenate Videos

Kyoto Cabinet and Kyoto Tycoon

• Setup and Install Kyoto Cabinet
• Setup and Install Kyoto Tycoon
• KT's RPC API Reference

Linux Administration

• Reset Compiz to Default Settings
• Enable and Start sendmail
• Installing sendmail
• Sending Emails using sendmail
• Find and Mark Bad Blocks
• Using TLP to optimize battery lifetime

MAC OSX

• Asynchronous Execution
  • NSThread
  • performSelectorInBackground
  • dispatch_async
• MacPorts

$ sudo port selfupdate
$ sudo port upgrade outdated
$ sudo port uninstall inactive
$ (just once) sudo port install port_cutleaves
$ sudo port_cutleaves

MongoDB

• Installation on Ubuntu
• Troubleshooting on Desktop
• Use in PHP via PECL driver

MySQL

• Error 1045: Access denied
• The insert ... on duplicate key update Pitfall
• Shrink ibfile1 InnoDB File
• Index JSON columns

PDF

• Merging PDF Files using GhostScript
• Cutting PDF Files using GhostScript
• Converting to PDF using GhostScript

PHP

• Convert Accented Characters to Non-Accented
• Upload a File using cURL
• HTTP Post in Pure PHP (without cURL)
• Prevent Client Disconnect from Killing PHP
• Disconnect Client and Continue Processing

PSQL, Postgres, PostgreSQL

• Dump and Restore
• Delete Duplicate Rows
• Set Sequence Value to Max of Table's ID

Python

• Timing, and memory, on Linux
• Importing Files
• uWSGI, nginx, Flask
• Decorators
• virtualenv
• SQLAlchemy
• iPython fails with NameError: name 'sys' is not defined on autocompletion

Security

• Password Generators
• SSH Access using pubkey's Authentication
• Using RSync together with SSH

Subversion/SVN

• Simple Guide to SVN Client
• Creating a SVN Repository
• Moving a SVN Repository
• Change Repository URL (Subversion 1.7+)
• Setup a SVN Server on a Shared Hosting with no root Access
• creating and applying patch (the --diff-cmd is necessary in case you set some graphical diff-viewer in ~/.subversion/config):

cd root_of_project
svn di --diff-cmd diff > ~/patch.diff
cd root_of_other_project
patch -p0 -i ~/patch.diff

• svn:externals - pull repo1 into dir1, and repo2 into dir2:

echo -e 'dir1 svn://repo1\ndir2 svn://repo2' > repos.txt
svn propset svn:externals . -F repos.txt

validate with " svn propget svn:externals . " .

Ubuntu / Debian / Gnome etc.

• restrict gnome app switcher to current workspace only:

gsettings set org.gnome.shell.app-switcher current-workspace-only true

• restore or undelete (config) file or directory:

# 1. check what package file or folder belongs to
$ dpkg -S /path/my.conf
# 2.a replace the config file; if it exists, will offer diff
$ sudo apt-get -o Dpkg::Options::="--force-confask" install --reinstall <package-name>
# 2.b restore the directory
$ sudo apt-get -o Dpkg::Options::="--force-confask" install --reinstall $(dpkg -S /etc/some/directory | sed 's/,//g; s/:.*//')

• increase number of inotify watchers

$ cat /proc/sys/fs/inotify/max_user_watches 
8192
$ echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf
$ sudo sysctl -p
$ cat /proc/sys/fs/inotify/max_user_watches 
524288

Up-to-Date Versions of Packages

• Python

Ubuntu 18.04 ships with Python 3.6, but Amazon Linux uses 3.7, so e.g. to use and build Lambda layers and match those against packages you deploy, you need to install 3.7:

$ sudo add-apt-repository ppa:deadsnakes/ppa
$ sudo apt update
$ sudo apt install python3.7 python3.7-venv

Since these do not override default python and python3 links (and better not!), you should create corresponding venv's as:

$ python3.7 -m venv [env-dir]

• NPM, node.js, etc.

The default apt npm version is awfully outdated.

update - available as a snap as well, no need to add apt sources - https://node.dev/node-binary

Add NodeSource APT repo; for other versions, e.g. 12.x, change to setup_12.x.

$ curl -sL https://deb.nodesource.com/setup_10.x | sudo -E bash -
$ sudo apt update
$ sudo apt install nodejs

ViM

• Delete all lines matching a pattern
• Directly type HTML entities
• Reformat a plain text paragraph

$ vim -x filename                      - open filename for editing, and save it encrypted (will prompt for a key)
:X                                     - set encryption for current file; use ':set key=' to disable encryption
                                       - put "set cm=blowfish" in your .vimrc to enable safer encryption by default
$ vim +10 [file] ; vim +/hello [file]  - open [file] at line 10, or at first occurrence of pattern 'hello' in file
:w !sudo tee %                         - save file via `sudo` (as root)

Wavelets

• Integer Haar Wavelets in Python

Web - HTML, CSS, JS, jQuery, etc.

• Multiple Submit Buttons and/or Multiple Forms on Page
• Make 'float' Elements Appear One under Another
• Upload a File via Form
• Override the IE's Problem with width
• Make an Element Fill In All the Remaining Space

Other Short Notes

HTTP Response Codes

This is not a definitive list, it's just stuff I use the most, and always forget which is which.

• 100 - Continue
• 200 - OK
• 201 - Created
• 202 - Accepted
• 204 - No Content
• 301 - Moved Permanently
• 302 - Found (I typically use this instead of the 307 "Temporary Redirect")
• 304 - Not Modified
• 400 - Bad Request
• 401 - Unauthorized
• 403 - Forbidden
• 404 - Not Found
• 405 - Method Not Allowed (using GET on POST calls, etc. - must include "Allow" response header)
• 406 - Not Acceptable (mismatch of response format and "Accept" headers)
• 500 - Internal Server Error
• 501 - Not Implemented
• 503 - Service Unavailable