PaskvilWiki - User contributions [en]

Short Notes on PSQL

2021-09-17T10:50:05Z

Admin: /* Set Sequence Value to Max of Table's ID */

PSQL aka Postgres aka PostgreSQL... I prefer psql.

== Dump and Restore ==

<pre>dump:
$ pg_dump -h host -p 5432 -U user -F c -b -v -f /tmp/db_name.backup db_name
restore:
$ pg_restore -h host -p 5432 -U user -d db_name -v /tmp/db_name.backup</pre>

== Delete Duplicate Lines ==

Before you can add a <tt>unique</tt> constraint to a table, you have to make sure it does satisfy this criteria.

With <tt>table_T</tt> and columns <tt>criteria_1</tt>, ..., <tt>criteria_N</tt>.

<pre>
;;
;; list rows that do not satisfy the uniqueness constraint
;;
SELECT
criteria_1,
...
criteria_N,
COUNT(*)
FROM
table_T
GROUP BY
criteria_1, ..., criteria_N
HAVING
COUNT(*) > 1
ORDER BY
criteria_1, ..., criteria_N;

;;
;; delete all rows that do not satisfy the constraint, keeping the ones with lowest id value
;;
DELETE FROM
table_T a
USING table_T b
WHERE
a.id > b.id
AND a.criteria_1 = b.criteria_1
...
AND a.criteria_N = b.criteria_N;
</pre>

== Set Sequence Value to Max of Table's ID ==

<pre>
SELECT setval('table_id_seq', (SELECT MAX(id) FROM table));
</pre>

== Find and Kill Stuck Queries ==

To get list of queries that are running for more than 5 minutes:
<pre>
SELECT
pid,
NOW() - pg_stat_activity.query_start AS duration,
query,
state
FROM pg_stat_activity
WHERE (NOW() - pg_stat_activity.query_start) > INTERVAL '5 MINUTES';
</pre>

Kill these by PID:
<pre>
SELECT pg_terminate_backend(_pid_);
</pre>

Short Notes on Python

2021-09-15T16:09:56Z

Admin: /* Subqueries */

== Miscellaneous ==

=== Date, Time, Datetime ===

<pre>d: datetime.date = ...
t: datetime.time = ...
dt: datetime.datetime = ...

dt.date() -> datetime.date
dt.time() -> datetime.time
datetime.datetime.combine(d, t) -> datetime.datetime
in particular: datetime.datetime.combine(d, datetime.time(0, 0, 0)) -> start of date `d` as datetime</pre>

=== <tt>strftime()</tt> Formatting ===

Examples for ''Thu Dec 3, 2020, 01:23:45.678'' - <tt>datetime.datetime(2020, 12, 3, 1, 23, 45, 678000)</tt>

<pre>
%y - year (20)
%Y - year (2020)
%m / %-m - month zero-/un- padded (12 / 12)
%d / %-d - day zero-/un- padded (03 / 3)
%H / %-H - hour zero-/un- padded, 24-hour clock (01 / 1)
%I / %-I - hour zero-/un- padded, 12-hour clock (01 / 1)
%p - time of day (AM)
%M / %-M - minute zero-/un- padded (23 / 23)
%S / %-S - second zero-/un- padded (45 / 45)
%f - microsecond (678000)
%z - timezone as +HHMM or -HHMM, empty if object is naive
%Z - timezone name, empty if object is naive
%j / %-j - day of year zero-/un- padded (338 / 338)
%a / %A - weekday name, abb. / full (Thu / Thursday)
%w - weekday as decimal, 0-Sunday, 6-Saturday (4)
%U / %W - week number, week starting on Sunday / Monday (48 / 48)
%b / %B - month name, abb. / full (Dec / December)
%c - locale appropriate date and time representation (Thu Dec 3 01:23:45 2020)
%x - locale appropriate date representation (12/03/20)
%X - locale appropriate time representation (01:23:45)
</pre>

=== Memory ===

For me, the following does a good job getting memory usage (in kB) on Linux:
<pre>import resource
print resource.getrusage(resource.RUSAGE_SELF).ru_maxrss</pre>
Since <tt>resource</tt> is standard package, it should work on Windows too, but I don't know if it does, or what units are used if it works.

== Importing Files ==

If you need to import a file '../mylib/commons.py', you can use the following snippet:
<nowiki>import sys, os
fld = os.path.realpath(os.path.abspath(os.path.join('..', 'mylib')))
if fld not in sys.path:
sys.path.insert(0, fld)
import commons

# use your commons.py module now...
commons.super_function()</nowiki>

== Merging Dictionaries ==

Below are few solutions, the first one works for any list of dictionaries, the rest is just for 2 dict's, with the last 2 suitable for inlining:

<pre># as a generic function that merges list of dict's
def merge_dicts(dicts: list) -> dict:
res = dict()
for d in dicts:
res.update(d)
return res

# if you're merging known number of dict's:

def merge_two_1(a: dict, b: dict) -> dict:
return dict(a, **b)

# python 3.5+, the fastest of the lot
def merge_two_1(a: dict, b: dict) -> dict:
return {**a, **b}</pre>

== uWSGI, nginx, Flask ==

* install uwsgi (incl. uwsgi python plugin), python flask, and nginx,

=== Setting Up uWSGI ===

* create ''main.py'' file that will hold the server logic, for instance:
<pre>from flask import Flask

app = Flask(__name__)

@app.route("/")
def hello():
return "hello there!"</pre>
* create uwsgi config file, ''wsgi.ini'' ('''minimal''' version here; read uwsgi docs for head-spinning array of configurables):
<pre>[uwsgi]
module = main:app
master = true
processes = 5
socket = 127.0.0.1:8000
protocol = http
plugin = python</pre>
* run uwsgi
uwsgi --ini wsgi.ini
* check that all works on http://localhost:8000/

=== Adding nginx Layer ===

* remove the "protocol" directive from ''wsgi.ini'', and add "die-on-term":
<pre>[uwsgi]
module = main:app
master = true
processes = 5
socket = 127.0.0.1:8000
plugin = python
die-on-term = true</pre>
* add a new vhost to nginx - ''/etc/nginx/sites-available/app.nginx'':
<pre>server {
listen 80;
server_name my.awesome.domain;
location / {
include uwsgi_params;
uwsgi_pass 127.0.0.1:8000;
}
}</pre>
** communication through socket is also possible (see ''socket'', ''chmod-socket'', ''vacuum'' and other directives for uWSGI)
** of course, create link in ''/etc/nginx/sites-enabled/'', and restart nginx,

=== Run uWSGI daemon on boot - supervisor ===

* install <tt>supervisor</tt>
apt-get install supervisor
* add/edit <tt>/etc/supervisord.conf</tt> with content like this:
<pre>[unix_http_server]
file=/run/supervisor.sock

[supervisord]
logfile=/var/log/supervisord.log
logfile_maxbytes=1MB
logfile_backups=1
loglevel=info ; (others: warn,debug,trace)
pidfile=/run/supervisord.pid
nodaemon=false
minfds=1024
minprocs=200

[supervisorctl]
serverurl=unix:///run/supervisor.sock

[program:your app]
directory=/path/to/app
command=/usr/bin/uwsgi --ini wsgi.ini
autostart=true
autorestart=true
stdout_logfile=/var/log/uwsgi.log
redirect_stderr=true
stopsignal=QUIT</pre>

=== Run uWSGI daemon on boot - systemd ===

* create ''systemd'' file for uWSGI, ''/etc/systemd/system/uwsgi-app.service'':
<pre>[Unit]
Description=Job that runs the uWSGI app

[Service]
Type=simple
WorkingDirectory=/home/project/flask-test/
ExecStart=/usr/bin/uwsgi --ini wsgi.ini

[Install]
WantedBy=multi-user.target</pre>

Then you can start and stop the uwsgi service using:
# systemctl start uwsgi-app.service
# systemctl stop uwsgi-app.service

Once you're happy with the settings, ''enable'' the daemon to be run on boot:
# systemctl enable uwsgi-app.service

== Decorators ==

''needs a bit of refreshment and updating...''

[http://www.artima.com/weblogs/viewpost.jsp?thread=240845 based on].

Decorators are simple and expressive way to modify function without editing the source of the function itself. Or, the other way around, to modify multiple functions in the same way, without code duplication (DRY).

Decorators can be spotted in the code by starting with <tt>@</tt> character. Decorator is a function (or class) that can do some additional work before or after the call to the decorated function. It can even call the decorated function multiple times, or not at all.

Decorators can be implemented as closures (my fave), or as classes; the following approaches are equivalent:

<pre># using a function (closure) as a decorator
def beforeAndAfter(f):
def decorated_fn():
print("Before", f.__name__)
f()
print("After", f.__name__)
return decorated_fn

@entryExit
def func():
print "func() is in da' house!"

func()</pre>

<pre># using a class as a decorator
class beforeAndAfter(object):

def __init__(self, f):
self.f = f

def __call__(self):
print("Before", self.f.__name__)
self.f()
print("After", self.f.__name__)

@beforeAndAfter
def func():
print "func() is in da' house!"

func()</pre>

Note that in functional decorator, anything outside the body of <tt>decorated_fn()</tt> is equivalent to content of the <tt>__init__()</tt> constructor of the class-based decorator. This code is run during the initialization, only once for each decorated function, regardless of whether the decorated function is ever called in the code - you should avoid any heavy lifting there.

Decorators can also accept arguments, and (obviously should) forward arguments to the decorated function:

<pre>def beforeAndAfter(p1, p2):
def wrap(f):
def wrapped_f(*args, **kwargs):
print "Decorator arguments:", p1, p2
print("Before", f.__name__)
f(*args, **kwargs)
print("After", f.__name__)
return wrapped_f
return wrap

@beforeAndAfter("hello", "world")
def func(a, b=2):
print "func() is in da' house,", a, b

func(1)</pre>

== virtualenv ==

I prefer installing <tt>virtualenv</tt> tool through <tt>pip</tt>, to make sure those are in sync version-wise.

<pre># create a new venv
# it is better to keep venv's in some separate folder, not to pollute your project folder
$ virtualenv ~/.virtualenvs/my-new-env

# "log into" your venv; success can be seen by your command line being prefixed by venv name
$ source ~/.virtualenvs/my-new-env/bin/activate

# now you can install anything you need, tucked away in your venv:
(my-new-env) $ pip install -r requirements.txt

# when done, just deactivate
(my-new-env) $ deactivate</pre>

== SQLAlchemy ==

=== Select, Where, Group ===

=== Updating ===

<pre>
session.query(Table).\
filter(Table.id.in_(...)).\
update({
Table.column1: 10,
Table.column2: Table.column2 + 50
})
</pre>

=== Scalar Values ===

==== <tt>min</tt>, <tt>max</tt>, ... ====

<pre>
from sqlalchemy import func

max_value = session.query(sqlalchemy.func.max(Table.column)).scalar()

# or, for multiple, just use
session.query(sqlalchemy.func.min(Table.column), sqlalchemy.func.max(Table.column)).first()
</pre>

==== <tt>select count(*)</tt> ====

<pre>session.query(MyModel).with_entities(func.count()).scalar()</pre>

=== Make SQLAlchemy Read-Only ===

All write operations in SQLAlchemy pass through <tt>flush()</tt> method of your session.
Just monkey-path it to do nothing!
<pre>engine = create_engine("connection string")
Session = sessionmaker(bind=engine, autoflush=False, autocommit=False)
session = Session()
session.flush = lambda *args,**kwargs: None</pre>

=== Making Session Self-Recoverable ===

Some DB's, most prominently Postgres, do not recover well from errors (e.g. Pg just keeps saying that there's error in current transaction).
One way is to rollback the transaction on "any" error:

<pre>from contextlib import contextmanager
from typing import ContextManager
from sqlalchemy.orm.session import Session

@contextmanager
def db_session() -> ContextManager[Session]:
session = init() # get the session any way you like
try:
yield session
except:
session.rollback()
raise
else:
session.commit()

def use_db():
with db_session() as session:
# use session object</pre>

=== Show Create Statement for Your Model, including Indexes ===

<pre>from sqlalchemy.schema import CreateTable, CreateIndex
from sqlalchemy.dialects import postgresql

def print_create_statement(model):
print(CreateTable(model.__table__).compile(dialect=postgresql.dialect()))
for index in model.__table__.indexes:
print(CreateIndex(index).compile(dialect=postgresql.dialect()))</pre>

=== List Tables and Their Columns ===

<pre>import sqlalchemy as sqla

db_conn_str = "..your connection string.."
engine = sqla.create_engine(db_conn_str)
inspector = sqla.inspect(engine)
schemas = inspector.get_schema_names()

for schema in schemas:
print("schema: {}".format(schema))
for table_name in inspector.get_table_names(schema=schema):
cols = ['{} ({})'.format(col['name'], col['type']) for col in inspector.get_columns(table_name, schema=schema)]
print("Table: {} ({})".format(table_name, ', '.join(cols)))</pre>

=== Subqueries ===

Creating subqueries is pretty straightforward in SQLAlchemy.

To emit
<pre>
SELECT sub.c1, sub.c2
FROM (SELECT x as c1, y as c2 FROM table) AS sub
</pre>
you define the <tt>sub</tt> and then query on it as usual, with the exception of bit more explicit access to columns:
<pre>
sub = session.query(table.x.label('c1'),
table.y.label('c2')).\
cte('sub')

res = session.query(sub.c.c1, sub.c.c2).\
all()
</pre>

=== Aliases ===

Aliases are super useful when you e.g. need to select something twice.

For instance, having "Book" records that have "author" and "editor" columns pointing to "User" table:
<pre>
from sqlalchemy.orm import aliased

author = aliased(User)
editor = aliased(User)
book_data = session.query(Book, author.name, editor.name).\
select_from(Book).\
join(author, Book.author_id == author.id).\
join(editor, Book.editor_id == editor.id).\
all()
</pre>

== iPython fails with ''NameError: name 'sys' is not defined'' on autocompletion ==

This is due to new version of <tt>jedi</tt> module not being compatible with iPython.

Just downgrade it to pre-0.18 version, using
<pre>
pip install --upgrade 'jedi<0.18.0'
</pre>

Main Page

2021-07-09T15:34:24Z

Admin: /* PSQL, Postgres, PostgreSQL */

Welcome to the '''PaskvilWiki'''!

The future shape of this wiki is unknown for now, let's hope it'll be something good...

This is partially a private wiki - viewing is allowed to all, but editing is allowed only to registered users.

This is more of a prevention than denial - I don't have time to monitor and moderate edits; - you're more than welcome to request registration, all contributors are very much welcome!

Just write me at josef.at.paskvil.dot.com.

== References and Tutorials ==

=== [[emacs and slime reference|Emacs and Slime Reference]] ===

=== [[Small jQuery Reference]] ===

=== [[Small jQuery Cheat Sheet]] ===

=== [[Small SPARQL, RDQL, etc. Cheat Sheet]] ===

== Short Notes - Tips and Tricks ==

=== [[Small 4store How-To|4store]] ===

* setup and installation
* running 4store
* data import
* data import and querying from Python
* Turtle file printer (in Python)

=== [[Short Notes on Apache|Apache]] ===

* Note on serving of local files
* Self-Signed SSL Certificate
* Enable SSL/HTTPS in Apache
* <tt>.htaccess</tt> and <tt>mod_rewrite</tt> Tricks

=== [[Short Notes on AWS|AWS]] ===

* Debugging (lack of) connection to instance
* Authorization header being removed by ElasticBeanstalk
* Notes on IAM
* AWS Lambda with Python - Notes
* boto3 Snippets
* AWS Chalice
* Building a Python Layer

=== [[Short Notes on Bash|Bash]] ===

* Extensions and File Name
* File Test Operators
* ''Commands and Shortcuts''
<pre>$ sudo !! - run last command as root
$ !!:gs/foo/bar - run last command, replacing 'foo' with 'bar'
$ ^foo^bar - run last command, replacing 'foo' with 'bar'
$ cp filename{,.bak} - create a .bak copy of file
<Ctrl-x> <Ctrl-e> - open $EDITOR to edit current command line; upon exit, content will be run
$ <space>command - run the command without saving it in history
$ > file.txt - empty a file
<Alt+.> - put last argument on cursor place; repeat to cycle through
$ (cd /tmp && ls) - jump to /tmp, execute ls, and jump back here
$ pushd /tmp .... $ popd - goes to /tmp, remembering current location; at later time, popd will bring you back
$ \command - run command without using aliases
$ rm !(pattern|pattern) - delete files that do not match given patterns
$ find . -mtime +5 -exec rm {} \; - delete all files in this folder that are older than 5 days
<Ctrl-u> ... <Ctrl-y> - Ctrl+u will cut current command, then you can do whatever you need, and Ctrl+y will paste it back
$ date -d @1234567890 - convert unix timestamp to human-readable
$ jul () { date -d "$1-01-01 +$2 days -1 day" "+%Y%m%d"; } - `jul $year $daynum` prints $daynum's day of $year, with Jan 1st being day 1</pre>
* ''Informations''
<pre>$ mount | column -t - pretty-print mount table
$ cat /etc/issue - display distro name</pre>
* ''Small Tricks''
<pre>$ echo $(printf '%q' "A\"B") - prints escaped string (in this case, > A\"B <)
$ mount -t tmpfs tmpfs /mnt -o size=1024m - mount temporary RAM partition (fast access, non-persistent!)
$ disown -a && exit - exit the shell while keeping all running processes alive
$ ssh -t remote_host screen -r - directly connect to a remote screen process
$ for I in $(mysql -e 'show databases' -s --skip-column-names); do mysqldump $I | gzip > "$I.sql.gz"; done
- backup all databases into individual files
$ fuser -k filename - kill process that is locking given file
$ readom dev=/dev/scd0 f=/path/to/image.iso - create CD/DVD ISO image
$ diff -rq folder-1 folder-2 - compare 2 folders (files missing or differing)
$ mkdir -p a/deep/dir/tree - create nested directories fast
$ mount file.iso /mnt/iso -o loop - mount ISO file as drive /mnt/iso (dir has to preexist)
$ sudo touch /forcefsck - force a file system check on the next reboot
$ ls | while read f; do process_file "$f"; done - iterate through files even if they contain space in name
$ while read f; do process_file "$f"; done < <(ls) - iterates through files, but does not create sub-process (outside scope is visible)</pre>
* ''lsof''
<pre>$ lsof -nPi - list network connections
$ netstat -tlnp - print all listening ports with PID's (run as root to see all process names)
$ lsof -c gnome-terminal - list files opened by given command
$ lsof +D [folder] - discover all open files in the [folder]
$ lsof -p [pid] - list files opened by given PID
$ lsof -i :8080 - list programs that have port 8080 open ("who to kill" if you get "port already in use")
$ if [ `lsof -- file | wc -l` -ne 0 ] ... - tests if file is open by some process</pre>

=== [[Short Notes on C/C++|C/C++]] ===

* How to clean up after child thread
* Non-blocking IO using sockets
* Shared Libraries on Linux+GCC
* Singleton
* <tt>std::move</tt> Semantics
* Converting between <tt>std::string</tt> and <tt>std::wstring</tt> in C++11

=== cURL (in terminal) ===

Use '''-s''' option to make <tt>curl</tt> silent.

* POST to server using cURL:
<pre>curl --data "key1=value1&key2=value2" http://www.example.com/post.php</pre>
* Upload to FTP server (with credentials), having connection timeout 60 seconds, and 120 seconds maximum operation time:
<pre>curl --connect-timeout 60 -m 120 -T file.ext "ftp://12.34.56.78/folder" --user user:pass</pre>
* Download from FTP server (with credentials):
<pre>curl --user user:pass "ftp://12.34.56.78/folder/file.ext" -o ~/Downloads/file.ext</pre>

=== [[Small Docker Notes|Docker]] ===

=== [[Small elasticsearch Notes|ElasticSearch]] ===

* Installation
* What You Get
* Indexing, Routing, etc.
* Deleting
* Getting and Multi-Getting
* Updating
* Search and Query DSL

=== [[Short Notes on ffmpeg|ffmpeg]] ===

* ''process all files, may include spaces in file name:''
ls *.avi | while read f; do ffmpeg -i "$f" ''...params...'' "$f.mkv"; done
* ''if you're having problems with input format, passing video through mpeg typically helps:''
ffmpeg -i file.mkv -qscale 2 -f mpeg - | ffmpeg -i - ''...params...'' file.avi
* ''downmix 5.1 dolby to normal stereo:''
ffmpeg -i file.mkv ''...params...'' -acodec libmp3lame -ar 44100 -ac 2 file.avi
* ''add an overlay image to the video:''
ffmpeg -i input.mp4 -i image.png -filter_complex "[0:v][1:v] overlay=''x'':''y'':enable='between(t,0,20)'" -pix_fmt yuv420p -c:a copy output.mp4
This will add ''image.png'' as an overlay at position ''x'', ''y'' over the ''imput.mp4'' video, for the first 20 seconds.
* ''fade in and out, first and last 25 frames of a 1000 frame video''
ffmpeg -i input.mp4 -vf "fade=in:0:25,fade=out:975:25" -acodec copy out.mp4
* ''convert album contained in APE or similar file with CUE into separate MP3 tracks''
ffmpeg -i cd.ape -ab 256k cd.mp3
mp3splt -a -c cd.cue cd.mp3

==== [[Short Notes on ffmpeg|more on ffmpeg]] ====

* Rip DVD into FLV (or anything else)
* Video File with Still Image and Audio
* Concatenate Videos

=== [[Short Notes on Flask and Flask-RestPlus|Flask and Flask-RestPlus]] ===

=== [[Short Notes on git|git]] ===

=== [[Short Notes on JS|All things JS - React, Vue, ...]] ===

=== [[Small Kyoto Cabinet and Kyoto Tycoon Notes|Kyoto Cabinet and Kyoto Tycoon]] ===

* Setup and Install Kyoto Cabinet
* Setup and Install Kyoto Tycoon
* KT's RPC API Reference

=== [[Short Notes on Linux Administration|Linux Administration]] ===

* Reset Compiz to Default Settings
* Enable and Start <tt>sendmail</tt>
* Installing <tt>sendmail</tt>
* Sending Emails using <tt>sendmail</tt>
* Find and Mark Bad Blocks
* Using TLP to optimize battery lifetime

=== [[Short Notes on MAC OSX|MAC OSX]] ===

* Asynchronous Execution
** ''NSThread''
** ''performSelectorInBackground''
** ''dispatch_async''
* ''MacPorts''
$ sudo port selfupdate
$ sudo port upgrade outdated
$ sudo port uninstall inactive
$ (just once) sudo port install port_cutleaves
$ sudo port_cutleaves

=== [[Small MongoDB Notes|MongoDB]] ===

* Installation on Ubuntu
* Troubleshooting on Desktop
* Use in PHP via PECL driver

=== [[Short Notes on MySQL|MySQL]] ===

* Error 1045: Access denied
* The <tt>insert ... on duplicate key update</tt> Pitfall
* Shrink ''ibfile1'' InnoDB File
* Index JSON columns

=== [[Short Notes on PDF|PDF]] ===

* Merging PDF Files using GhostScript
* Cutting PDF Files using GhostScript
* Converting to PDF using GhostScript

=== [[Short Notes on PHP|PHP]] ===

* Convert Accented Characters to Non-Accented
* Upload a File using cURL
* HTTP Post in Pure PHP (without cURL)
* Prevent Client Disconnect from Killing PHP
* Disconnect Client and Continue Processing

=== [[Short Notes on PSQL|PSQL, Postgres, PostgreSQL]] ===

* Dump and Restore
* Delete Duplicate Rows
* Set Sequence Value to Max of Table's ID

=== [[Short Notes on Python|Python]] ===

* Timing, and memory, on Linux
* Importing Files
* uWSGI, nginx, Flask
* Decorators
* virtualenv
* SQLAlchemy
* iPython fails with ''NameError: name 'sys' is not defined'' on autocompletion

=== [[RedBeanPHP Cheat Sheet]] ===

=== [[Short Notes on RESTful APIs|RESTful APIs]] ===

=== [[Short Notes on Security|Security]] ===

* Password Generators
* SSH Access using pubkey's Authentication
* Using RSync together with SSH

=== [[Short Notes on SVN|Subversion/SVN]] ===

* Simple Guide to SVN Client
* Creating a SVN Repository
* Moving a SVN Repository
* Change Repository URL (Subversion 1.7+)
* Setup a SVN Server on a Shared Hosting with no ''root'' Access
* creating and applying patch (the <tt>--diff-cmd</tt> is necessary in case you set some graphical diff-viewer in <tt>~/.subversion/config</tt>):
<pre>cd root_of_project
svn di --diff-cmd diff > ~/patch.diff
cd root_of_other_project
patch -p0 -i ~/patch.diff</pre>
* ''svn:externals'' - pull ''repo1'' into ''dir1'', and ''repo2'' into ''dir2'':
<pre>echo -e 'dir1 svn://repo1\ndir2 svn://repo2' > repos.txt
svn propset svn:externals . -F repos.txt</pre>
validate with " ''svn propget svn:externals .'' " .

=== Ubuntu / Debian / Gnome etc. ===

* restrict gnome app switcher to current workspace only:
<pre>gsettings set org.gnome.shell.app-switcher current-workspace-only true</pre>
* restore or undelete (config) file or directory:
<pre># 1. check what package file or folder belongs to
$ dpkg -S /path/my.conf
# 2.a replace the config file; if it exists, will offer diff
$ sudo apt-get -o Dpkg::Options::="--force-confask" install --reinstall <package-name>
# 2.b restore the directory
$ sudo apt-get -o Dpkg::Options::="--force-confask" install --reinstall $(dpkg -S /etc/some/directory | sed 's/,//g; s/:.*//')</pre>
* increase number of inotify watchers
<pre>$ cat /proc/sys/fs/inotify/max_user_watches
8192
$ echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf
$ sudo sysctl -p
$ cat /proc/sys/fs/inotify/max_user_watches
524288</pre>

==== Up-to-Date Versions of Packages ====

* '''Python'''
Ubuntu 18.04 ships with Python 3.6, but Amazon Linux uses 3.7, so e.g. to use and build Lambda layers and match those against packages you deploy, you need to install 3.7:
<pre>$ sudo add-apt-repository ppa:deadsnakes/ppa
$ sudo apt update
$ sudo apt install python3.7 python3.7-venv</pre>
Since these do not override default <tt>python</tt> and <tt>python3</tt> links (and better not!), you should create corresponding venv's as:
<pre>$ python3.7 -m venv [env-dir]</pre>

* '''NPM, node.js, etc.'''
The default <tt>apt</tt> ''npm'' version is awfully outdated.

'''update''' - available as a ''snap'' as well, no need to add ''apt'' sources - https://node.dev/node-binary

Add NodeSource APT repo; for other versions, e.g. 12.x, change to <tt>setup_12.x</tt>.
<pre>$ curl -sL https://deb.nodesource.com/setup_10.x | sudo -E bash -
$ sudo apt update
$ sudo apt install nodejs</pre>

=== [[Short Notes on ViM|ViM]] ===

* Delete all lines matching a pattern
* Directly type HTML entities
* Reformat a plain text paragraph
<pre>$ vim -x filename - open filename for editing, and save it encrypted (will prompt for a key)
:X - set encryption for current file; use ':set key=' to disable encryption
- put "set cm=blowfish" in your .vimrc to enable safer encryption by default
$ vim +10 [file] ; vim +/hello [file] - open [file] at line 10, or at first occurrence of pattern 'hello' in file
:w !sudo tee % - save file via `sudo` (as root)</pre>

=== [[Short Notes on Wavelets|Wavelets]] ===

* Integer Haar Wavelets in Python

=== [[Short Notes on Web|Web - HTML, CSS, JS, jQuery, etc.]] ===

* Multiple Submit Buttons and/or Multiple Forms on Page
* Make 'float' Elements Appear One under Another
* Upload a File via Form
* Override the IE's Problem with <tt>width</tt>
* Make an Element Fill In All the Remaining Space

=== Other Short Notes ===

==== [[Rip Video DVD that is not Finalized]] ====

==== [[Install Apache and PHP to do Secure h264 Pseudo Streaming]] ====

==== HTTP Response Codes ====

This is not a definitive list, it's just stuff I use the most, and always forget which is which.

* '''100''' - ''Continue''
* '''200''' - ''OK''
* '''201''' - ''Created''
* '''202''' - ''Accepted''
* '''204''' - ''No Content''
* '''301''' - ''Moved Permanently''
* '''302''' - ''Found'' (I typically use this instead of the '''307''' "''Temporary Redirect''")
* '''304''' - ''Not Modified''
* '''400''' - ''Bad Request''
* '''401''' - ''Unauthorized''
* '''403''' - ''Forbidden''
* '''404''' - ''Not Found''
* '''405''' - ''Method Not Allowed'' (using GET on POST calls, etc. - must include "Allow" response header)
* '''406''' - ''Not Acceptable'' (mismatch of response format and "Accept" headers)
* '''500''' - ''Internal Server Error''
* '''501''' - ''Not Implemented''
* '''503''' - ''Service Unavailable''

2021-02-20T12:16:55Z

Admin:

Main Page

2021-01-25T22:05:50Z

Admin: /* Up-to-Date Versions of Packages */

Welcome to the '''PaskvilWiki'''!

The future shape of this wiki is unknown for now, let's hope it'll be something good...

This is partially a private wiki - viewing is allowed to all, but editing is allowed only to registered users.

This is more of a prevention than denial - I don't have time to monitor and moderate edits; - you're more than welcome to request registration, all contributors are very much welcome!

Just write me at josef.at.paskvil.dot.com.

== References and Tutorials ==

=== [[emacs and slime reference|Emacs and Slime Reference]] ===

=== [[Small jQuery Reference]] ===

=== [[Small jQuery Cheat Sheet]] ===

=== [[Small SPARQL, RDQL, etc. Cheat Sheet]] ===

== Short Notes - Tips and Tricks ==

=== [[Small 4store How-To|4store]] ===

* setup and installation
* running 4store
* data import
* data import and querying from Python
* Turtle file printer (in Python)

=== [[Short Notes on Apache|Apache]] ===

* Note on serving of local files
* Self-Signed SSL Certificate
* Enable SSL/HTTPS in Apache
* <tt>.htaccess</tt> and <tt>mod_rewrite</tt> Tricks

=== [[Short Notes on AWS|AWS]] ===

* Debugging (lack of) connection to instance
* Authorization header being removed by ElasticBeanstalk
* Notes on IAM
* AWS Lambda with Python - Notes
* boto3 Snippets
* AWS Chalice
* Building a Python Layer

=== [[Short Notes on Bash|Bash]] ===

* Extensions and File Name
* File Test Operators
* ''Commands and Shortcuts''
<pre>$ sudo !! - run last command as root
$ !!:gs/foo/bar - run last command, replacing 'foo' with 'bar'
$ ^foo^bar - run last command, replacing 'foo' with 'bar'
$ cp filename{,.bak} - create a .bak copy of file
<Ctrl-x> <Ctrl-e> - open $EDITOR to edit current command line; upon exit, content will be run
$ <space>command - run the command without saving it in history
$ > file.txt - empty a file
<Alt+.> - put last argument on cursor place; repeat to cycle through
$ (cd /tmp && ls) - jump to /tmp, execute ls, and jump back here
$ pushd /tmp .... $ popd - goes to /tmp, remembering current location; at later time, popd will bring you back
$ \command - run command without using aliases
$ rm !(pattern|pattern) - delete files that do not match given patterns
$ find . -mtime +5 -exec rm {} \; - delete all files in this folder that are older than 5 days
<Ctrl-u> ... <Ctrl-y> - Ctrl+u will cut current command, then you can do whatever you need, and Ctrl+y will paste it back
$ date -d @1234567890 - convert unix timestamp to human-readable
$ jul () { date -d "$1-01-01 +$2 days -1 day" "+%Y%m%d"; } - `jul $year $daynum` prints $daynum's day of $year, with Jan 1st being day 1</pre>
* ''Informations''
<pre>$ mount | column -t - pretty-print mount table
$ cat /etc/issue - display distro name</pre>
* ''Small Tricks''
<pre>$ echo $(printf '%q' "A\"B") - prints escaped string (in this case, > A\"B <)
$ mount -t tmpfs tmpfs /mnt -o size=1024m - mount temporary RAM partition (fast access, non-persistent!)
$ disown -a && exit - exit the shell while keeping all running processes alive
$ ssh -t remote_host screen -r - directly connect to a remote screen process
$ for I in $(mysql -e 'show databases' -s --skip-column-names); do mysqldump $I | gzip > "$I.sql.gz"; done
- backup all databases into individual files
$ fuser -k filename - kill process that is locking given file
$ readom dev=/dev/scd0 f=/path/to/image.iso - create CD/DVD ISO image
$ diff -rq folder-1 folder-2 - compare 2 folders (files missing or differing)
$ mkdir -p a/deep/dir/tree - create nested directories fast
$ mount file.iso /mnt/iso -o loop - mount ISO file as drive /mnt/iso (dir has to preexist)
$ sudo touch /forcefsck - force a file system check on the next reboot
$ ls | while read f; do process_file "$f"; done - iterate through files even if they contain space in name
$ while read f; do process_file "$f"; done < <(ls) - iterates through files, but does not create sub-process (outside scope is visible)</pre>
* ''lsof''
<pre>$ lsof -nPi - list network connections
$ netstat -tlnp - print all listening ports with PID's (run as root to see all process names)
$ lsof -c gnome-terminal - list files opened by given command
$ lsof +D [folder] - discover all open files in the [folder]
$ lsof -p [pid] - list files opened by given PID
$ lsof -i :8080 - list programs that have port 8080 open ("who to kill" if you get "port already in use")
$ if [ `lsof -- file | wc -l` -ne 0 ] ... - tests if file is open by some process</pre>

=== [[Short Notes on C/C++|C/C++]] ===

* How to clean up after child thread
* Non-blocking IO using sockets
* Shared Libraries on Linux+GCC
* Singleton
* <tt>std::move</tt> Semantics
* Converting between <tt>std::string</tt> and <tt>std::wstring</tt> in C++11

=== cURL (in terminal) ===

Use '''-s''' option to make <tt>curl</tt> silent.

* POST to server using cURL:
<pre>curl --data "key1=value1&key2=value2" http://www.example.com/post.php</pre>
* Upload to FTP server (with credentials), having connection timeout 60 seconds, and 120 seconds maximum operation time:
<pre>curl --connect-timeout 60 -m 120 -T file.ext "ftp://12.34.56.78/folder" --user user:pass</pre>
* Download from FTP server (with credentials):
<pre>curl --user user:pass "ftp://12.34.56.78/folder/file.ext" -o ~/Downloads/file.ext</pre>

=== [[Small Docker Notes|Docker]] ===

=== [[Small elasticsearch Notes|ElasticSearch]] ===

* Installation
* What You Get
* Indexing, Routing, etc.
* Deleting
* Getting and Multi-Getting
* Updating
* Search and Query DSL

=== [[Short Notes on ffmpeg|ffmpeg]] ===

* ''process all files, may include spaces in file name:''
ls *.avi | while read f; do ffmpeg -i "$f" ''...params...'' "$f.mkv"; done
* ''if you're having problems with input format, passing video through mpeg typically helps:''
ffmpeg -i file.mkv -qscale 2 -f mpeg - | ffmpeg -i - ''...params...'' file.avi
* ''downmix 5.1 dolby to normal stereo:''
ffmpeg -i file.mkv ''...params...'' -acodec libmp3lame -ar 44100 -ac 2 file.avi
* ''add an overlay image to the video:''
ffmpeg -i input.mp4 -i image.png -filter_complex "[0:v][1:v] overlay=''x'':''y'':enable='between(t,0,20)'" -pix_fmt yuv420p -c:a copy output.mp4
This will add ''image.png'' as an overlay at position ''x'', ''y'' over the ''imput.mp4'' video, for the first 20 seconds.
* ''fade in and out, first and last 25 frames of a 1000 frame video''
ffmpeg -i input.mp4 -vf "fade=in:0:25,fade=out:975:25" -acodec copy out.mp4
* ''convert album contained in APE or similar file with CUE into separate MP3 tracks''
ffmpeg -i cd.ape -ab 256k cd.mp3
mp3splt -a -c cd.cue cd.mp3

==== [[Short Notes on ffmpeg|more on ffmpeg]] ====

* Rip DVD into FLV (or anything else)
* Video File with Still Image and Audio
* Concatenate Videos

=== [[Short Notes on Flask and Flask-RestPlus|Flask and Flask-RestPlus]] ===

=== [[Short Notes on git|git]] ===

=== [[Short Notes on JS|All things JS - React, Vue, ...]] ===

=== [[Small Kyoto Cabinet and Kyoto Tycoon Notes|Kyoto Cabinet and Kyoto Tycoon]] ===

* Setup and Install Kyoto Cabinet
* Setup and Install Kyoto Tycoon
* KT's RPC API Reference

=== [[Short Notes on Linux Administration|Linux Administration]] ===

* Reset Compiz to Default Settings
* Enable and Start <tt>sendmail</tt>
* Installing <tt>sendmail</tt>
* Sending Emails using <tt>sendmail</tt>
* Find and Mark Bad Blocks
* Using TLP to optimize battery lifetime

=== [[Short Notes on MAC OSX|MAC OSX]] ===

* Asynchronous Execution
** ''NSThread''
** ''performSelectorInBackground''
** ''dispatch_async''
* ''MacPorts''
$ sudo port selfupdate
$ sudo port upgrade outdated
$ sudo port uninstall inactive
$ (just once) sudo port install port_cutleaves
$ sudo port_cutleaves

=== [[Small MongoDB Notes|MongoDB]] ===

* Installation on Ubuntu
* Troubleshooting on Desktop
* Use in PHP via PECL driver

=== [[Short Notes on MySQL|MySQL]] ===

* Error 1045: Access denied
* The <tt>insert ... on duplicate key update</tt> Pitfall

=== [[Short Notes on PDF|PDF]] ===

* Merging PDF Files using GhostScript
* Cutting PDF Files using GhostScript
* Converting to PDF using GhostScript

=== [[Short Notes on PHP|PHP]] ===

* Convert Accented Characters to Non-Accented
* Upload a File using cURL
* HTTP Post in Pure PHP (without cURL)
* Prevent Client Disconnect from Killing PHP
* Disconnect Client and Continue Processing

=== [[Short Notes on PSQL|PSQL, Postgres, PostgreSQL]] ===

=== [[Short Notes on Python|Python]] ===

* Timing, and memory, on Linux
* Importing Files
* uWSGI, nginx, Flask
* Decorators
* virtualenv
* SQLAlchemy

=== [[RedBeanPHP Cheat Sheet]] ===

=== [[Short Notes on RESTful APIs|RESTful APIs]] ===

=== [[Short Notes on Security|Security]] ===

* Password Generators
* SSH Access using pubkey's Authentication
* Using RSync together with SSH

=== [[Short Notes on SVN|Subversion/SVN]] ===

* Simple Guide to SVN Client
* Creating a SVN Repository
* Moving a SVN Repository
* Change Repository URL (Subversion 1.7+)
* Setup a SVN Server on a Shared Hosting with no ''root'' Access
* creating and applying patch (the <tt>--diff-cmd</tt> is necessary in case you set some graphical diff-viewer in <tt>~/.subversion/config</tt>):
<pre>cd root_of_project
svn di --diff-cmd diff > ~/patch.diff
cd root_of_other_project
patch -p0 -i ~/patch.diff</pre>
* ''svn:externals'' - pull ''repo1'' into ''dir1'', and ''repo2'' into ''dir2'':
<pre>echo -e 'dir1 svn://repo1\ndir2 svn://repo2' > repos.txt
svn propset svn:externals . -F repos.txt</pre>
validate with " ''svn propget svn:externals .'' " .

=== Ubuntu / Debian / Gnome etc. ===

* restrict gnome app switcher to current workspace only:
<pre>gsettings set org.gnome.shell.app-switcher current-workspace-only true</pre>
* restore or undelete (config) file or directory:
<pre># 1. check what package file or folder belongs to
$ dpkg -S /path/my.conf
# 2.a replace the config file; if it exists, will offer diff
$ sudo apt-get -o Dpkg::Options::="--force-confask" install --reinstall <package-name>
# 2.b restore the directory
$ sudo apt-get -o Dpkg::Options::="--force-confask" install --reinstall $(dpkg -S /etc/some/directory | sed 's/,//g; s/:.*//')</pre>
* increase number of inotify watchers
<pre>$ cat /proc/sys/fs/inotify/max_user_watches
8192
$ echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf
$ sudo sysctl -p
$ cat /proc/sys/fs/inotify/max_user_watches
524288</pre>

==== Up-to-Date Versions of Packages ====

* '''Python'''
Ubuntu 18.04 ships with Python 3.6, but Amazon Linux uses 3.7, so e.g. to use and build Lambda layers and match those against packages you deploy, you need to install 3.7:
<pre>$ sudo add-apt-repository ppa:deadsnakes/ppa
$ sudo apt update
$ sudo apt install python3.7 python3.7-venv</pre>
Since these do not override default <tt>python</tt> and <tt>python3</tt> links (and better not!), you should create corresponding venv's as:
<pre>$ python3.7 -m venv [env-dir]</pre>

* '''NPM, node.js, etc.'''
The default <tt>apt</tt> ''npm'' version is awfully outdated.

'''update''' - available as a ''snap'' as well, no need to add ''apt'' sources - https://node.dev/node-binary

Add NodeSource APT repo; for other versions, e.g. 12.x, change to <tt>setup_12.x</tt>.
<pre>$ curl -sL https://deb.nodesource.com/setup_10.x | sudo -E bash -
$ sudo apt update
$ sudo apt install nodejs</pre>

=== [[Short Notes on ViM|ViM]] ===

* Delete all lines matching a pattern
* Directly type HTML entities
* Reformat a plain text paragraph
<pre>$ vim -x filename - open filename for editing, and save it encrypted (will prompt for a key)
:X - set encryption for current file; use ':set key=' to disable encryption
- put "set cm=blowfish" in your .vimrc to enable safer encryption by default
$ vim +10 [file] ; vim +/hello [file] - open [file] at line 10, or at first occurrence of pattern 'hello' in file
:w !sudo tee % - save file via `sudo` (as root)</pre>

=== [[Short Notes on Wavelets|Wavelets]] ===

* Integer Haar Wavelets in Python

=== [[Short Notes on Web|Web - HTML, CSS, JS, jQuery, etc.]] ===

* Multiple Submit Buttons and/or Multiple Forms on Page
* Make 'float' Elements Appear One under Another
* Upload a File via Form
* Override the IE's Problem with <tt>width</tt>
* Make an Element Fill In All the Remaining Space

=== Other Short Notes ===

==== [[Rip Video DVD that is not Finalized]] ====

==== [[Install Apache and PHP to do Secure h264 Pseudo Streaming]] ====

==== HTTP Response Codes ====

This is not a definitive list, it's just stuff I use the most, and always forget which is which.

* '''100''' - ''Continue''
* '''200''' - ''OK''
* '''201''' - ''Created''
* '''202''' - ''Accepted''
* '''204''' - ''No Content''
* '''301''' - ''Moved Permanently''
* '''302''' - ''Found'' (I typically use this instead of the '''307''' "''Temporary Redirect''")
* '''304''' - ''Not Modified''
* '''400''' - ''Bad Request''
* '''401''' - ''Unauthorized''
* '''403''' - ''Forbidden''
* '''404''' - ''Not Found''
* '''405''' - ''Method Not Allowed'' (using GET on POST calls, etc. - must include "Allow" response header)
* '''406''' - ''Not Acceptable'' (mismatch of response format and "Accept" headers)
* '''500''' - ''Internal Server Error''
* '''501''' - ''Not Implemented''
* '''503''' - ''Service Unavailable''

Short Notes on JS

2020-12-11T20:12:38Z

Admin:

== Simple Mapping ==

<pre>
let a = {a: 1, b: 2, c: 3};

Object.keys(a).map(i => console.log(i));
> a
> b
> c
>> [undefined, undefined, undefined]

a = [1, 2, 3];
a.map((i, k) => console.log(i, k));
> 1 0
> 2 1
> 3 2
>> [undefined, undefined, undefined]

// fill() is required here, as it materializes the Array;
// otherwise, Array(8) would be treated as empty array by map()
Array(8).fill().map((_, i) => i * i);
>> [0, 1, 4, 9, 16, 25, 36, 49]

== Fetch Patterns ==

<pre>fetch("url", { options... })
.then((response) => {
if (!response.ok)
throw new Error('Network response was not ok');
return response.json(); // or response.blob(), etc.
})
.then((data) => {
// do something with the data received
})
.catch((error) => {
console.error('Failed to fetch:', error);
});</pre>

== Environment Variables in WebPack ==

WebPack does not have access to environment (duh), so you need to "bake" any relevant environment variables in the WebPack during build:
<pre>new webpack.DefinePlugin({
'process.env': {
NODE_ENV: JSON.stringify(process.env.NODE_ENV),
STAGE: JSON.stringify(process.env.STAGE),
// ...
}
})</pre>

== Debugging in VS Code on Linux ==

* install ''Debugger for Chrome'' extension
* open the ''launch.json'' file, and edit as follows:
** custom profile - ''user-data-dir'' - is used, to make sure you have clean (enough) slate
<pre>{
// Use IntelliSense to learn about possible attributes.
// Hover to view descriptions of existing attributes.
// For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
"version": "0.2.0",
"configurations": [
{
"type": "chrome",
"request": "launch",
"runtimeExecutable": "/usr/bin/chromium-browser",
"runtimeArgs": ["--remote-debugging-port=9222", "--user-data-dir=/home/user/tmp/remote-profile"],
"name": "Launch Chrome against localhost",
"url": "http://localhost:3000",
"webRoot": "${workspaceFolder}"
}
]
}</pre>
* run the npm (the port used should correspond to the ''url'' key above)
* hit debug in VS Code, add breakpoints, etc., and enjoy!
** this will actually open new browser window, as subwindow of VS Code, neat!

Short Notes on Python

2020-12-04T13:01:41Z

Admin: /* SQLAlchemy */

Short Notes on Python

2020-12-03T11:55:16Z

Admin:

== Miscellaneous ==

=== Date, Time, Datetime ===

<pre>d: datetime.date = ...
t: datetime.time = ...
dt: datetime.datetime = ...

dt.date() -> datetime.date
dt.time() -> datetime.time
datetime.datetime.combine(d, t) -> datetime.datetime
in particular: datetime.datetime.combine(d, datetime.time(0, 0, 0)) -> start of date `d` as datetime</pre>

=== <tt>strftime()</tt> Formatting ===

Examples for ''Thu Dec 3, 2020, 01:23:45.678'' - <tt>datetime.datetime(2020, 12, 3, 1, 23, 45, 678000)</tt>

<pre>
%y - year (20)
%Y - year (2020)
%m / %-m - month zero-/un- padded (12 / 12)
%d / %-d - day zero-/un- padded (03 / 3)
%H / %-H - hour zero-/un- padded, 24-hour clock (01 / 1)
%I / %-I - hour zero-/un- padded, 12-hour clock (01 / 1)
%p - time of day (AM)
%M / %-M - minute zero-/un- padded (23 / 23)
%S / %-S - second zero-/un- padded (45 / 45)
%f - microsecond (678000)
%z - timezone as +HHMM or -HHMM, empty if object is naive
%Z - timezone name, empty if object is naive
%j / %-j - day of year zero-/un- padded (338 / 338)
%a / %A - weekday name, abb. / full (Thu / Thursday)
%w - weekday as decimal, 0-Sunday, 6-Saturday (4)
%U / %W - week number, week starting on Sunday / Monday (48 / 48)
%b / %B - month name, abb. / full (Dec / December)
%c - locale appropriate date and time representation (Thu Dec 3 01:23:45 2020)
%x - locale appropriate date representation (12/03/20)
%X - locale appropriate time representation (01:23:45)
</pre>

=== Memory ===

For me, the following does a good job getting memory usage (in kB) on Linux:
<pre>import resource
print resource.getrusage(resource.RUSAGE_SELF).ru_maxrss</pre>
Since <tt>resource</tt> is standard package, it should work on Windows too, but I don't know if it does, or what units are used if it works.

== Importing Files ==

If you need to import a file '../mylib/commons.py', you can use the following snippet:
<nowiki>import sys, os
fld = os.path.realpath(os.path.abspath(os.path.join('..', 'mylib')))
if fld not in sys.path:
sys.path.insert(0, fld)
import commons

# use your commons.py module now...
commons.super_function()</nowiki>

== Merging Dictionaries ==

Below are few solutions, the first one works for any list of dictionaries, the rest is just for 2 dict's, with the last 2 suitable for inlining:

<pre># as a generic function that merges list of dict's
def merge_dicts(dicts: list) -> dict:
res = dict()
for d in dicts:
res.update(d)
return res

# if you're merging known number of dict's:

def merge_two_1(a: dict, b: dict) -> dict:
return dict(a, **b)

# python 3.5+, the fastest of the lot
def merge_two_1(a: dict, b: dict) -> dict:
return {**a, **b}</pre>

== uWSGI, nginx, Flask ==

* install uwsgi (incl. uwsgi python plugin), python flask, and nginx,

=== Setting Up uWSGI ===

* create ''main.py'' file that will hold the server logic, for instance:
<pre>from flask import Flask

app = Flask(__name__)

@app.route("/")
def hello():
return "hello there!"</pre>
* create uwsgi config file, ''wsgi.ini'' ('''minimal''' version here; read uwsgi docs for head-spinning array of configurables):
<pre>[uwsgi]
module = main:app
master = true
processes = 5
socket = 127.0.0.1:8000
protocol = http
plugin = python</pre>
* run uwsgi
uwsgi --ini wsgi.ini
* check that all works on http://localhost:8000/

=== Adding nginx Layer ===

* remove the "protocol" directive from ''wsgi.ini'', and add "die-on-term":
<pre>[uwsgi]
module = main:app
master = true
processes = 5
socket = 127.0.0.1:8000
plugin = python
die-on-term = true</pre>
* add a new vhost to nginx - ''/etc/nginx/sites-available/app.nginx'':
<pre>server {
listen 80;
server_name my.awesome.domain;
location / {
include uwsgi_params;
uwsgi_pass 127.0.0.1:8000;
}
}</pre>
** communication through socket is also possible (see ''socket'', ''chmod-socket'', ''vacuum'' and other directives for uWSGI)
** of course, create link in ''/etc/nginx/sites-enabled/'', and restart nginx,

=== Run uWSGI daemon on boot - supervisor ===

* install <tt>supervisor</tt>
apt-get install supervisor
* add/edit <tt>/etc/supervisord.conf</tt> with content like this:
<pre>[unix_http_server]
file=/run/supervisor.sock

[supervisord]
logfile=/var/log/supervisord.log
logfile_maxbytes=1MB
logfile_backups=1
loglevel=info ; (others: warn,debug,trace)
pidfile=/run/supervisord.pid
nodaemon=false
minfds=1024
minprocs=200

[supervisorctl]
serverurl=unix:///run/supervisor.sock

[program:your app]
directory=/path/to/app
command=/usr/bin/uwsgi --ini wsgi.ini
autostart=true
autorestart=true
stdout_logfile=/var/log/uwsgi.log
redirect_stderr=true
stopsignal=QUIT</pre>

=== Run uWSGI daemon on boot - systemd ===

* create ''systemd'' file for uWSGI, ''/etc/systemd/system/uwsgi-app.service'':
<pre>[Unit]
Description=Job that runs the uWSGI app

[Service]
Type=simple
WorkingDirectory=/home/project/flask-test/
ExecStart=/usr/bin/uwsgi --ini wsgi.ini

[Install]
WantedBy=multi-user.target</pre>

Then you can start and stop the uwsgi service using:
# systemctl start uwsgi-app.service
# systemctl stop uwsgi-app.service

Once you're happy with the settings, ''enable'' the daemon to be run on boot:
# systemctl enable uwsgi-app.service

== Decorators ==

''needs a bit of refreshment and updating...''

[http://www.artima.com/weblogs/viewpost.jsp?thread=240845 based on].

Decorators are simple and expressive way to modify function without editing the source of the function itself. Or, the other way around, to modify multiple functions in the same way, without code duplication (DRY).

Decorators can be spotted in the code by starting with <tt>@</tt> character. Decorator is a function (or class) that can do some additional work before or after the call to the decorated function. It can even call the decorated function multiple times, or not at all.

Decorators can be implemented as closures (my fave), or as classes; the following approaches are equivalent:

<pre># using a function (closure) as a decorator
def beforeAndAfter(f):
def decorated_fn():
print("Before", f.__name__)
f()
print("After", f.__name__)
return decorated_fn

@entryExit
def func():
print "func() is in da' house!"

func()</pre>

<pre># using a class as a decorator
class beforeAndAfter(object):

def __init__(self, f):
self.f = f

def __call__(self):
print("Before", self.f.__name__)
self.f()
print("After", self.f.__name__)

@beforeAndAfter
def func():
print "func() is in da' house!"

func()</pre>

Note that in functional decorator, anything outside the body of <tt>decorated_fn()</tt> is equivalent to content of the <tt>__init__()</tt> constructor of the class-based decorator. This code is run during the initialization, only once for each decorated function, regardless of whether the decorated function is ever called in the code - you should avoid any heavy lifting there.

Decorators can also accept arguments, and (obviously should) forward arguments to the decorated function:

<pre>def beforeAndAfter(p1, p2):
def wrap(f):
def wrapped_f(*args, **kwargs):
print "Decorator arguments:", p1, p2
print("Before", f.__name__)
f(*args, **kwargs)
print("After", f.__name__)
return wrapped_f
return wrap

@beforeAndAfter("hello", "world")
def func(a, b=2):
print "func() is in da' house,", a, b

func(1)</pre>

== virtualenv ==

I prefer installing <tt>virtualenv</tt> tool through <tt>pip</tt>, to make sure those are in sync version-wise.

<pre># create a new venv
# it is better to keep venv's in some separate folder, not to pollute your project folder
$ virtualenv ~/.virtualenvs/my-new-env

# "log into" your venv; success can be seen by your command line being prefixed by venv name
$ source ~/.virtualenvs/my-new-env/bin/activate

# now you can install anything you need, tucked away in your venv:
(my-new-env) $ pip install -r requirements.txt

# when done, just deactivate
(my-new-env) $ deactivate</pre>

== SQLAlchemy ==

=== Some Basics ===

==== Select, Where, Group ====

==== Updating ====

<pre>
session.query(Table).\
filter(Table.id.in_(...)).\
update({
Table.column1: 10,
Table.column2: Table.column2 + 50
})
</pre>

==== Scalar Values ====

<pre>
max_value = session.query(sqlalchemy.func.max(Table.column)).scalar()

# or, for multiple, just use
session.query(sqlalchemy.func.min(Table.column), sqlalchemy.func.max(Table.column)).first()
</pre>

=== Make SQLAlchemy Read-Only ===

All write operations in SQLAlchemy pass through <tt>flush()</tt> method of your session.
Just monkey-path it to do nothing!
<pre>engine = create_engine("connection string")
Session = sessionmaker(bind=engine, autoflush=False, autocommit=False)
session = Session()
session.flush = lambda *args,**kwargs: None</pre>

=== Making Session Self-Recoverable ===

Some DB's, most prominently Postgres, do not recover well from errors (e.g. Pg just keeps saying that there's error in current transaction).
One way is to rollback the transaction on "any" error:

<pre>from contextlib import contextmanager
from typing import ContextManager
from sqlalchemy.orm.session import Session

@contextmanager
def db_session() -> ContextManager[Session]:
session = init() # get the session any way you like
try:
yield session
except:
session.rollback()
raise
else:
session.commit()

def use_db():
with db_session() as session:
# use session object</pre>

=== Show Create Statement for Your Model, including Indexes ===

<pre>from sqlalchemy.schema import CreateTable, CreateIndex
from sqlalchemy.dialects import postgresql

def print_create_statement(model):
print(CreateTable(model.__table__).compile(dialect=postgresql.dialect()))
for index in model.__table__.indexes:
print(CreateIndex(index).compile(dialect=postgresql.dialect()))</pre>

=== List Tables and Their Columns ===

<pre>import sqlalchemy as sqla

db_conn_str = "..your connection string.."
engine = sqla.create_engine(db_conn_str)
inspector = sqla.inspect(engine)
schemas = inspector.get_schema_names()

for schema in schemas:
print("schema: {}".format(schema))
for table_name in inspector.get_table_names(schema=schema):
cols = ['{} ({})'.format(col['name'], col['type']) for col in inspector.get_columns(table_name, schema=schema)]
print("Table: {} ({})".format(table_name, ', '.join(cols)))</pre>

=== <tt>select count(*)</tt> ===

<pre>session.query(MyModel).with_entities(func.count()).scalar()</pre>

=== Subqueries ===

Creating subqueries is pretty straightforward in SQLAlchemy.

To emit
<pre>
SELECT sub.c1, sub.c2
FROM (SELECT x as c1, y as c2 FROM table) AS sub
</pre>
you define the <tt>sub</tt> and then query on it as usual, with the exception of bit more explicit access to columns:
<pre>
sub = session.query(table.x.label('c1'),
table.y.label('c2')).\
cte('sub')

res = session.query(sub.c.c1, sub.c.c2).\
all()
</pre>

Short Notes on Python

2020-12-03T01:53:06Z

Admin:

== Miscellaneous ==

=== Date, Time, Datetime ===

<pre>d: datetime.date = ...
t: datetime.time = ...
dt: datetime.datetime = ...

dt.date() -> datetime.date
dt.time() -> datetime.time
datetime.datetime.combine(d, t) -> datetime.datetime
in particular: datetime.datetime.combine(d, datetime.time(0, 0, 0)) -> start of date `d` as datetime</pre>

=== Memory ===

For me, the following does a good job getting memory usage (in kB) on Linux:
<pre>import resource
print resource.getrusage(resource.RUSAGE_SELF).ru_maxrss</pre>
Since <tt>resource</tt> is standard package, it should work on Windows too, but I don't know if it does, or what units are used if it works.

== Importing Files ==

If you need to import a file '../mylib/commons.py', you can use the following snippet:
<nowiki>import sys, os
fld = os.path.realpath(os.path.abspath(os.path.join('..', 'mylib')))
if fld not in sys.path:
sys.path.insert(0, fld)
import commons

# use your commons.py module now...
commons.super_function()</nowiki>

== Merging Dictionaries ==

Below are few solutions, the first one works for any list of dictionaries, the rest is just for 2 dict's, with the last 2 suitable for inlining:

<pre># as a generic function that merges list of dict's
def merge_dicts(dicts: list) -> dict:
res = dict()
for d in dicts:
res.update(d)
return res

# if you're merging known number of dict's:

def merge_two_1(a: dict, b: dict) -> dict:
return dict(a, **b)

# python 3.5+, the fastest of the lot
def merge_two_1(a: dict, b: dict) -> dict:
return {**a, **b}</pre>

== uWSGI, nginx, Flask ==

* install uwsgi (incl. uwsgi python plugin), python flask, and nginx,

=== Setting Up uWSGI ===

* create ''main.py'' file that will hold the server logic, for instance:
<pre>from flask import Flask

app = Flask(__name__)

@app.route("/")
def hello():
return "hello there!"</pre>
* create uwsgi config file, ''wsgi.ini'' ('''minimal''' version here; read uwsgi docs for head-spinning array of configurables):
<pre>[uwsgi]
module = main:app
master = true
processes = 5
socket = 127.0.0.1:8000
protocol = http
plugin = python</pre>
* run uwsgi
uwsgi --ini wsgi.ini
* check that all works on http://localhost:8000/

=== Adding nginx Layer ===

* remove the "protocol" directive from ''wsgi.ini'', and add "die-on-term":
<pre>[uwsgi]
module = main:app
master = true
processes = 5
socket = 127.0.0.1:8000
plugin = python
die-on-term = true</pre>
* add a new vhost to nginx - ''/etc/nginx/sites-available/app.nginx'':
<pre>server {
listen 80;
server_name my.awesome.domain;
location / {
include uwsgi_params;
uwsgi_pass 127.0.0.1:8000;
}
}</pre>
** communication through socket is also possible (see ''socket'', ''chmod-socket'', ''vacuum'' and other directives for uWSGI)
** of course, create link in ''/etc/nginx/sites-enabled/'', and restart nginx,

=== Run uWSGI daemon on boot - supervisor ===

* install <tt>supervisor</tt>
apt-get install supervisor
* add/edit <tt>/etc/supervisord.conf</tt> with content like this:
<pre>[unix_http_server]
file=/run/supervisor.sock

[supervisord]
logfile=/var/log/supervisord.log
logfile_maxbytes=1MB
logfile_backups=1
loglevel=info ; (others: warn,debug,trace)
pidfile=/run/supervisord.pid
nodaemon=false
minfds=1024
minprocs=200

[supervisorctl]
serverurl=unix:///run/supervisor.sock

[program:your app]
directory=/path/to/app
command=/usr/bin/uwsgi --ini wsgi.ini
autostart=true
autorestart=true
stdout_logfile=/var/log/uwsgi.log
redirect_stderr=true
stopsignal=QUIT</pre>

=== Run uWSGI daemon on boot - systemd ===

* create ''systemd'' file for uWSGI, ''/etc/systemd/system/uwsgi-app.service'':
<pre>[Unit]
Description=Job that runs the uWSGI app

[Service]
Type=simple
WorkingDirectory=/home/project/flask-test/
ExecStart=/usr/bin/uwsgi --ini wsgi.ini

[Install]
WantedBy=multi-user.target</pre>

Then you can start and stop the uwsgi service using:
# systemctl start uwsgi-app.service
# systemctl stop uwsgi-app.service

Once you're happy with the settings, ''enable'' the daemon to be run on boot:
# systemctl enable uwsgi-app.service

== Decorators ==

''needs a bit of refreshment and updating...''

[http://www.artima.com/weblogs/viewpost.jsp?thread=240845 based on].

Decorators are simple and expressive way to modify function without editing the source of the function itself. Or, the other way around, to modify multiple functions in the same way, without code duplication (DRY).

Decorators can be spotted in the code by starting with <tt>@</tt> character. Decorator is a function (or class) that can do some additional work before or after the call to the decorated function. It can even call the decorated function multiple times, or not at all.

Decorators can be implemented as closures (my fave), or as classes; the following approaches are equivalent:

<pre># using a function (closure) as a decorator
def beforeAndAfter(f):
def decorated_fn():
print("Before", f.__name__)
f()
print("After", f.__name__)
return decorated_fn

@entryExit
def func():
print "func() is in da' house!"

func()</pre>

<pre># using a class as a decorator
class beforeAndAfter(object):

def __init__(self, f):
self.f = f

def __call__(self):
print("Before", self.f.__name__)
self.f()
print("After", self.f.__name__)

@beforeAndAfter
def func():
print "func() is in da' house!"

func()</pre>

Note that in functional decorator, anything outside the body of <tt>decorated_fn()</tt> is equivalent to content of the <tt>__init__()</tt> constructor of the class-based decorator. This code is run during the initialization, only once for each decorated function, regardless of whether the decorated function is ever called in the code - you should avoid any heavy lifting there.

Decorators can also accept arguments, and (obviously should) forward arguments to the decorated function:

<pre>def beforeAndAfter(p1, p2):
def wrap(f):
def wrapped_f(*args, **kwargs):
print "Decorator arguments:", p1, p2
print("Before", f.__name__)
f(*args, **kwargs)
print("After", f.__name__)
return wrapped_f
return wrap

@beforeAndAfter("hello", "world")
def func(a, b=2):
print "func() is in da' house,", a, b

func(1)</pre>

== virtualenv ==

I prefer installing <tt>virtualenv</tt> tool through <tt>pip</tt>, to make sure those are in sync version-wise.

<pre># create a new venv
# it is better to keep venv's in some separate folder, not to pollute your project folder
$ virtualenv ~/.virtualenvs/my-new-env

# "log into" your venv; success can be seen by your command line being prefixed by venv name
$ source ~/.virtualenvs/my-new-env/bin/activate

# now you can install anything you need, tucked away in your venv:
(my-new-env) $ pip install -r requirements.txt

# when done, just deactivate
(my-new-env) $ deactivate</pre>

== SQLAlchemy ==

=== Some Basics ===

==== Select, Where, Group ====

==== Updating ====

<pre>
session.query(Table).\
filter(Table.id.in_(...)).\
update({
Table.column1: 10,
Table.column2: Table.column2 + 50
})
</pre>

==== Scalar Values ====

<pre>
max_value = session.query(sqlalchemy.func.max(Table.column)).scalar()

# or, for multiple, just use
session.query(sqlalchemy.func.min(Table.column), sqlalchemy.func.max(Table.column)).first()
</pre>

=== Make SQLAlchemy Read-Only ===

All write operations in SQLAlchemy pass through <tt>flush()</tt> method of your session.
Just monkey-path it to do nothing!
<pre>engine = create_engine("connection string")
Session = sessionmaker(bind=engine, autoflush=False, autocommit=False)
session = Session()
session.flush = lambda *args,**kwargs: None</pre>

=== Making Session Self-Recoverable ===

Some DB's, most prominently Postgres, do not recover well from errors (e.g. Pg just keeps saying that there's error in current transaction).
One way is to rollback the transaction on "any" error:

<pre>from contextlib import contextmanager
from typing import ContextManager
from sqlalchemy.orm.session import Session

@contextmanager
def db_session() -> ContextManager[Session]:
session = init() # get the session any way you like
try:
yield session
except:
session.rollback()
raise
else:
session.commit()

def use_db():
with db_session() as session:
# use session object</pre>

=== Show Create Statement for Your Model, including Indexes ===

<pre>from sqlalchemy.schema import CreateTable, CreateIndex
from sqlalchemy.dialects import postgresql

def print_create_statement(model):
print(CreateTable(model.__table__).compile(dialect=postgresql.dialect()))
for index in model.__table__.indexes:
print(CreateIndex(index).compile(dialect=postgresql.dialect()))</pre>

=== List Tables and Their Columns ===

<pre>import sqlalchemy as sqla

db_conn_str = "..your connection string.."
engine = sqla.create_engine(db_conn_str)
inspector = sqla.inspect(engine)
schemas = inspector.get_schema_names()

for schema in schemas:
print("schema: {}".format(schema))
for table_name in inspector.get_table_names(schema=schema):
cols = ['{} ({})'.format(col['name'], col['type']) for col in inspector.get_columns(table_name, schema=schema)]
print("Table: {} ({})".format(table_name, ', '.join(cols)))</pre>

=== <tt>select count(*)</tt> ===

<pre>session.query(MyModel).with_entities(func.count()).scalar()</pre>

=== Subqueries ===

Creating subqueries is pretty straightforward in SQLAlchemy.

To emit
<pre>
SELECT sub.c1, sub.c2
FROM (SELECT x as c1, y as c2 FROM table) AS sub
</pre>
you define the <tt>sub</tt> and then query on it as usual, with the exception of bit more explicit access to columns:
<pre>
sub = session.query(table.x.label('c1'),
table.y.label('c2')).\
cte('sub')

res = session.query(sub.c.c1, sub.c.c2).\
all()
</pre>

2020-04-08T15:46:30Z

Admin: Created page with "PSQL aka Postgres aka PostgreSQL... I prefer psql. == Dump and Restore == <pre>dump: $ pg_dump -h host -p 5432 -U user -F c -b -v -f /tmp/db_name.backup db_name restore: $ p..."

Main Page

2020-04-08T15:42:36Z

Admin: /* PHP */

Welcome to the '''PaskvilWiki'''!

The future shape of this wiki is unknown for now, let's hope it'll be something good...

This is partially a private wiki - viewing is allowed to all, but editing is allowed only to registered users.

This is more of a prevention than denial - I don't have time to monitor and moderate edits; - you're more than welcome to request registration, all contributors are very much welcome!

Just write me at josef.at.paskvil.dot.com.

== References and Tutorials ==

=== [[emacs and slime reference|Emacs and Slime Reference]] ===

=== [[Small jQuery Reference]] ===

=== [[Small jQuery Cheat Sheet]] ===

=== [[Small SPARQL, RDQL, etc. Cheat Sheet]] ===

== Short Notes - Tips and Tricks ==

=== [[Small 4store How-To|4store]] ===

* setup and installation
* running 4store
* data import
* data import and querying from Python
* Turtle file printer (in Python)

=== [[Short Notes on Apache|Apache]] ===

* Note on serving of local files
* Self-Signed SSL Certificate
* Enable SSL/HTTPS in Apache
* <tt>.htaccess</tt> and <tt>mod_rewrite</tt> Tricks

=== [[Short Notes on AWS|AWS]] ===

* Debugging (lack of) connection to instance
* Authorization header being removed by ElasticBeanstalk
* Notes on IAM
* AWS Lambda with Python - Notes
* boto3 Snippets
* AWS Chalice
* Building a Python Layer

=== [[Short Notes on Bash|Bash]] ===

* Extensions and File Name
* File Test Operators
* ''Commands and Shortcuts''
<pre>$ sudo !! - run last command as root
$ !!:gs/foo/bar - run last command, replacing 'foo' with 'bar'
$ ^foo^bar - run last command, replacing 'foo' with 'bar'
$ cp filename{,.bak} - create a .bak copy of file
<Ctrl-x> <Ctrl-e> - open $EDITOR to edit current command line; upon exit, content will be run
$ <space>command - run the command without saving it in history
$ > file.txt - empty a file
<Alt+.> - put last argument on cursor place; repeat to cycle through
$ (cd /tmp && ls) - jump to /tmp, execute ls, and jump back here
$ pushd /tmp .... $ popd - goes to /tmp, remembering current location; at later time, popd will bring you back
$ \command - run command without using aliases
$ rm !(pattern|pattern) - delete files that do not match given patterns
$ find . -mtime +5 -exec rm {} \; - delete all files in this folder that are older than 5 days
<Ctrl-u> ... <Ctrl-y> - Ctrl+u will cut current command, then you can do whatever you need, and Ctrl+y will paste it back
$ date -d @1234567890 - convert unix timestamp to human-readable
$ jul () { date -d "$1-01-01 +$2 days -1 day" "+%Y%m%d"; } - `jul $year $daynum` prints $daynum's day of $year, with Jan 1st being day 1</pre>
* ''Informations''
<pre>$ mount | column -t - pretty-print mount table
$ cat /etc/issue - display distro name</pre>
* ''Small Tricks''
<pre>$ echo $(printf '%q' "A\"B") - prints escaped string (in this case, > A\"B <)
$ mount -t tmpfs tmpfs /mnt -o size=1024m - mount temporary RAM partition (fast access, non-persistent!)
$ disown -a && exit - exit the shell while keeping all running processes alive
$ ssh -t remote_host screen -r - directly connect to a remote screen process
$ for I in $(mysql -e 'show databases' -s --skip-column-names); do mysqldump $I | gzip > "$I.sql.gz"; done
- backup all databases into individual files
$ fuser -k filename - kill process that is locking given file
$ readom dev=/dev/scd0 f=/path/to/image.iso - create CD/DVD ISO image
$ diff -rq folder-1 folder-2 - compare 2 folders (files missing or differing)
$ mkdir -p a/deep/dir/tree - create nested directories fast
$ mount file.iso /mnt/iso -o loop - mount ISO file as drive /mnt/iso (dir has to preexist)
$ sudo touch /forcefsck - force a file system check on the next reboot
$ ls | while read f; do process_file "$f"; done - iterate through files even if they contain space in name
$ while read f; do process_file "$f"; done < <(ls) - iterates through files, but does not create sub-process (outside scope is visible)</pre>
* ''lsof''
<pre>$ lsof -nPi - list network connections
$ netstat -tlnp - print all listening ports with PID's (run as root to see all process names)
$ lsof -c gnome-terminal - list files opened by given command
$ lsof +D [folder] - discover all open files in the [folder]
$ lsof -p [pid] - list files opened by given PID
$ lsof -i :8080 - list programs that have port 8080 open ("who to kill" if you get "port already in use")
$ if [ `lsof -- file | wc -l` -ne 0 ] ... - tests if file is open by some process</pre>

=== [[Short Notes on C/C++|C/C++]] ===

* How to clean up after child thread
* Non-blocking IO using sockets
* Shared Libraries on Linux+GCC
* Singleton
* <tt>std::move</tt> Semantics
* Converting between <tt>std::string</tt> and <tt>std::wstring</tt> in C++11

=== cURL (in terminal) ===

Use '''-s''' option to make <tt>curl</tt> silent.

* POST to server using cURL:
<pre>curl --data "key1=value1&key2=value2" http://www.example.com/post.php</pre>
* Upload to FTP server (with credentials), having connection timeout 60 seconds, and 120 seconds maximum operation time:
<pre>curl --connect-timeout 60 -m 120 -T file.ext "ftp://12.34.56.78/folder" --user user:pass</pre>
* Download from FTP server (with credentials):
<pre>curl --user user:pass "ftp://12.34.56.78/folder/file.ext" -o ~/Downloads/file.ext</pre>

=== [[Small Docker Notes|Docker]] ===

=== [[Small elasticsearch Notes|ElasticSearch]] ===

* Installation
* What You Get
* Indexing, Routing, etc.
* Deleting
* Getting and Multi-Getting
* Updating
* Search and Query DSL

=== [[Short Notes on ffmpeg|ffmpeg]] ===

* ''process all files, may include spaces in file name:''
ls *.avi | while read f; do ffmpeg -i "$f" ''...params...'' "$f.mkv"; done
* ''if you're having problems with input format, passing video through mpeg typically helps:''
ffmpeg -i file.mkv -qscale 2 -f mpeg - | ffmpeg -i - ''...params...'' file.avi
* ''downmix 5.1 dolby to normal stereo:''
ffmpeg -i file.mkv ''...params...'' -acodec libmp3lame -ar 44100 -ac 2 file.avi
* ''add an overlay image to the video:''
ffmpeg -i input.mp4 -i image.png -filter_complex "[0:v][1:v] overlay=''x'':''y'':enable='between(t,0,20)'" -pix_fmt yuv420p -c:a copy output.mp4
This will add ''image.png'' as an overlay at position ''x'', ''y'' over the ''imput.mp4'' video, for the first 20 seconds.
* ''fade in and out, first and last 25 frames of a 1000 frame video''
ffmpeg -i input.mp4 -vf "fade=in:0:25,fade=out:975:25" -acodec copy out.mp4
* ''convert album contained in APE or similar file with CUE into separate MP3 tracks''
ffmpeg -i cd.ape -ab 256k cd.mp3
mp3splt -a -c cd.cue cd.mp3

==== [[Short Notes on ffmpeg|more on ffmpeg]] ====

* Rip DVD into FLV (or anything else)
* Video File with Still Image and Audio
* Concatenate Videos

=== [[Short Notes on Flask and Flask-RestPlus|Flask and Flask-RestPlus]] ===

=== [[Short Notes on git|git]] ===

=== [[Short Notes on JS|All things JS - React, Vue, ...]] ===

=== [[Small Kyoto Cabinet and Kyoto Tycoon Notes|Kyoto Cabinet and Kyoto Tycoon]] ===

* Setup and Install Kyoto Cabinet
* Setup and Install Kyoto Tycoon
* KT's RPC API Reference

=== [[Short Notes on Linux Administration|Linux Administration]] ===

* Reset Compiz to Default Settings
* Enable and Start <tt>sendmail</tt>
* Installing <tt>sendmail</tt>
* Sending Emails using <tt>sendmail</tt>
* Find and Mark Bad Blocks
* Using TLP to optimize battery lifetime

=== [[Short Notes on MAC OSX|MAC OSX]] ===

* Asynchronous Execution
** ''NSThread''
** ''performSelectorInBackground''
** ''dispatch_async''
* ''MacPorts''
$ sudo port selfupdate
$ sudo port upgrade outdated
$ sudo port uninstall inactive
$ (just once) sudo port install port_cutleaves
$ sudo port_cutleaves

=== [[Small MongoDB Notes|MongoDB]] ===

* Installation on Ubuntu
* Troubleshooting on Desktop
* Use in PHP via PECL driver

=== [[Short Notes on MySQL|MySQL]] ===

* Error 1045: Access denied
* The <tt>insert ... on duplicate key update</tt> Pitfall

=== [[Short Notes on PDF|PDF]] ===

* Merging PDF Files using GhostScript
* Cutting PDF Files using GhostScript
* Converting to PDF using GhostScript

=== [[Short Notes on PHP|PHP]] ===

* Convert Accented Characters to Non-Accented
* Upload a File using cURL
* HTTP Post in Pure PHP (without cURL)
* Prevent Client Disconnect from Killing PHP
* Disconnect Client and Continue Processing

=== [[Short Notes on PSQL|PSQL, Postgres, PostgreSQL]] ===

=== [[Short Notes on Python|Python]] ===

* Timing, and memory, on Linux
* Importing Files
* uWSGI, nginx, Flask
* Decorators
* virtualenv
* SQLAlchemy

=== [[RedBeanPHP Cheat Sheet]] ===

=== [[Short Notes on RESTful APIs|RESTful APIs]] ===

=== [[Short Notes on Security|Security]] ===

* Password Generators
* SSH Access using pubkey's Authentication
* Using RSync together with SSH

=== [[Short Notes on SVN|Subversion/SVN]] ===

* Simple Guide to SVN Client
* Creating a SVN Repository
* Moving a SVN Repository
* Change Repository URL (Subversion 1.7+)
* Setup a SVN Server on a Shared Hosting with no ''root'' Access
* creating and applying patch (the <tt>--diff-cmd</tt> is necessary in case you set some graphical diff-viewer in <tt>~/.subversion/config</tt>):
<pre>cd root_of_project
svn di --diff-cmd diff > ~/patch.diff
cd root_of_other_project
patch -p0 -i ~/patch.diff</pre>
* ''svn:externals'' - pull ''repo1'' into ''dir1'', and ''repo2'' into ''dir2'':
<pre>echo -e 'dir1 svn://repo1\ndir2 svn://repo2' > repos.txt
svn propset svn:externals . -F repos.txt</pre>
validate with " ''svn propget svn:externals .'' " .

=== Ubuntu / Debian / Gnome etc. ===

* restrict gnome app switcher to current workspace only:
<pre>gsettings set org.gnome.shell.app-switcher current-workspace-only true</pre>
* restore or undelete (config) file or directory:
<pre># 1. check what package file or folder belongs to
$ dpkg -S /path/my.conf
# 2.a replace the config file; if it exists, will offer diff
$ sudo apt-get -o Dpkg::Options::="--force-confask" install --reinstall <package-name>
# 2.b restore the directory
$ sudo apt-get -o Dpkg::Options::="--force-confask" install --reinstall $(dpkg -S /etc/some/directory | sed 's/,//g; s/:.*//')</pre>
* increase number of inotify watchers
<pre>$ cat /proc/sys/fs/inotify/max_user_watches
8192
$ echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf
$ sudo sysctl -p
$ cat /proc/sys/fs/inotify/max_user_watches
524288</pre>

==== Up-to-Date Versions of Packages ====

* '''Python'''
Ubuntu 18.04 ships with Python 3.6, but Amazon Linux uses 3.7, so e.g. to use and build Lambda layers and match those against packages you deploy, you need to install 3.7:
<pre>$ sudo add-apt-repository ppa:deadsnakes/ppa
$ sudo apt update
$ sudo apt install python3.7</pre>
Since these do not override default <tt>python</tt> and <tt>python3</tt> links (and better not!), you should create corresponding venv's as:
<pre>$ python3.7 -m venv [env-dir]</pre>

* '''NPM, node.js, etc.'''
The default <tt>apt</tt> ''npm'' version is awfully outdated.

Add NodeSource APT repo; for other versions, e.g. 12.x, change to <tt>setup_12.x</tt>.
<pre>$ curl -sL https://deb.nodesource.com/setup_10.x | sudo -E bash -
$ sudo apt update
$ sudo apt install nodejs</pre>

=== [[Short Notes on ViM|ViM]] ===

* Delete all lines matching a pattern
* Directly type HTML entities
* Reformat a plain text paragraph
<pre>$ vim -x filename - open filename for editing, and save it encrypted (will prompt for a key)
:X - set encryption for current file; use ':set key=' to disable encryption
- put "set cm=blowfish" in your .vimrc to enable safer encryption by default
$ vim +10 [file] ; vim +/hello [file] - open [file] at line 10, or at first occurrence of pattern 'hello' in file
:w !sudo tee % - save file via `sudo` (as root)</pre>

=== [[Short Notes on Wavelets|Wavelets]] ===

* Integer Haar Wavelets in Python

=== [[Short Notes on Web|Web - HTML, CSS, JS, jQuery, etc.]] ===

* Multiple Submit Buttons and/or Multiple Forms on Page
* Make 'float' Elements Appear One under Another
* Upload a File via Form
* Override the IE's Problem with <tt>width</tt>
* Make an Element Fill In All the Remaining Space

=== Other Short Notes ===

==== [[Rip Video DVD that is not Finalized]] ====

==== [[Install Apache and PHP to do Secure h264 Pseudo Streaming]] ====

==== HTTP Response Codes ====

This is not a definitive list, it's just stuff I use the most, and always forget which is which.

* '''100''' - ''Continue''
* '''200''' - ''OK''
* '''201''' - ''Created''
* '''202''' - ''Accepted''
* '''204''' - ''No Content''
* '''301''' - ''Moved Permanently''
* '''302''' - ''Found'' (I typically use this instead of the '''307''' "''Temporary Redirect''")
* '''304''' - ''Not Modified''
* '''400''' - ''Bad Request''
* '''401''' - ''Unauthorized''
* '''403''' - ''Forbidden''
* '''404''' - ''Not Found''
* '''405''' - ''Method Not Allowed'' (using GET on POST calls, etc. - must include "Allow" response header)
* '''406''' - ''Not Acceptable'' (mismatch of response format and "Accept" headers)
* '''500''' - ''Internal Server Error''
* '''501''' - ''Not Implemented''
* '''503''' - ''Service Unavailable''

Main Page

2020-04-01T12:40:59Z

Admin: /* Bash */

Welcome to the '''PaskvilWiki'''!

The future shape of this wiki is unknown for now, let's hope it'll be something good...

This is partially a private wiki - viewing is allowed to all, but editing is allowed only to registered users.

This is more of a prevention than denial - I don't have time to monitor and moderate edits; - you're more than welcome to request registration, all contributors are very much welcome!

Just write me at josef.at.paskvil.dot.com.

== References and Tutorials ==

=== [[emacs and slime reference|Emacs and Slime Reference]] ===

=== [[Small jQuery Reference]] ===

=== [[Small jQuery Cheat Sheet]] ===

=== [[Small SPARQL, RDQL, etc. Cheat Sheet]] ===

== Short Notes - Tips and Tricks ==

=== [[Small 4store How-To|4store]] ===

* setup and installation
* running 4store
* data import
* data import and querying from Python
* Turtle file printer (in Python)

=== [[Short Notes on Apache|Apache]] ===

* Note on serving of local files
* Self-Signed SSL Certificate
* Enable SSL/HTTPS in Apache
* <tt>.htaccess</tt> and <tt>mod_rewrite</tt> Tricks

=== [[Short Notes on AWS|AWS]] ===

* Debugging (lack of) connection to instance
* Authorization header being removed by ElasticBeanstalk
* Notes on IAM
* AWS Lambda with Python - Notes
* boto3 Snippets
* AWS Chalice
* Building a Python Layer

=== [[Short Notes on Bash|Bash]] ===

* Extensions and File Name
* File Test Operators
* ''Commands and Shortcuts''
<pre>$ sudo !! - run last command as root
$ !!:gs/foo/bar - run last command, replacing 'foo' with 'bar'
$ ^foo^bar - run last command, replacing 'foo' with 'bar'
$ cp filename{,.bak} - create a .bak copy of file
<Ctrl-x> <Ctrl-e> - open $EDITOR to edit current command line; upon exit, content will be run
$ <space>command - run the command without saving it in history
$ > file.txt - empty a file
<Alt+.> - put last argument on cursor place; repeat to cycle through
$ (cd /tmp && ls) - jump to /tmp, execute ls, and jump back here
$ pushd /tmp .... $ popd - goes to /tmp, remembering current location; at later time, popd will bring you back
$ \command - run command without using aliases
$ rm !(pattern|pattern) - delete files that do not match given patterns
$ find . -mtime +5 -exec rm {} \; - delete all files in this folder that are older than 5 days
<Ctrl-u> ... <Ctrl-y> - Ctrl+u will cut current command, then you can do whatever you need, and Ctrl+y will paste it back
$ date -d @1234567890 - convert unix timestamp to human-readable
$ jul () { date -d "$1-01-01 +$2 days -1 day" "+%Y%m%d"; } - `jul $year $daynum` prints $daynum's day of $year, with Jan 1st being day 1</pre>
* ''Informations''
<pre>$ mount | column -t - pretty-print mount table
$ cat /etc/issue - display distro name</pre>
* ''Small Tricks''
<pre>$ echo $(printf '%q' "A\"B") - prints escaped string (in this case, > A\"B <)
$ mount -t tmpfs tmpfs /mnt -o size=1024m - mount temporary RAM partition (fast access, non-persistent!)
$ disown -a && exit - exit the shell while keeping all running processes alive
$ ssh -t remote_host screen -r - directly connect to a remote screen process
$ for I in $(mysql -e 'show databases' -s --skip-column-names); do mysqldump $I | gzip > "$I.sql.gz"; done
- backup all databases into individual files
$ fuser -k filename - kill process that is locking given file
$ readom dev=/dev/scd0 f=/path/to/image.iso - create CD/DVD ISO image
$ diff -rq folder-1 folder-2 - compare 2 folders (files missing or differing)
$ mkdir -p a/deep/dir/tree - create nested directories fast
$ mount file.iso /mnt/iso -o loop - mount ISO file as drive /mnt/iso (dir has to preexist)
$ sudo touch /forcefsck - force a file system check on the next reboot
$ ls | while read f; do process_file "$f"; done - iterate through files even if they contain space in name
$ while read f; do process_file "$f"; done < <(ls) - iterates through files, but does not create sub-process (outside scope is visible)</pre>
* ''lsof''
<pre>$ lsof -nPi - list network connections
$ netstat -tlnp - print all listening ports with PID's (run as root to see all process names)
$ lsof -c gnome-terminal - list files opened by given command
$ lsof +D [folder] - discover all open files in the [folder]
$ lsof -p [pid] - list files opened by given PID
$ lsof -i :8080 - list programs that have port 8080 open ("who to kill" if you get "port already in use")
$ if [ `lsof -- file | wc -l` -ne 0 ] ... - tests if file is open by some process</pre>

=== [[Short Notes on C/C++|C/C++]] ===

* How to clean up after child thread
* Non-blocking IO using sockets
* Shared Libraries on Linux+GCC
* Singleton
* <tt>std::move</tt> Semantics
* Converting between <tt>std::string</tt> and <tt>std::wstring</tt> in C++11

=== cURL (in terminal) ===

Use '''-s''' option to make <tt>curl</tt> silent.

* POST to server using cURL:
<pre>curl --data "key1=value1&key2=value2" http://www.example.com/post.php</pre>
* Upload to FTP server (with credentials), having connection timeout 60 seconds, and 120 seconds maximum operation time:
<pre>curl --connect-timeout 60 -m 120 -T file.ext "ftp://12.34.56.78/folder" --user user:pass</pre>
* Download from FTP server (with credentials):
<pre>curl --user user:pass "ftp://12.34.56.78/folder/file.ext" -o ~/Downloads/file.ext</pre>

=== [[Small Docker Notes|Docker]] ===

=== [[Small elasticsearch Notes|ElasticSearch]] ===

* Installation
* What You Get
* Indexing, Routing, etc.
* Deleting
* Getting and Multi-Getting
* Updating
* Search and Query DSL

=== [[Short Notes on ffmpeg|ffmpeg]] ===

* ''process all files, may include spaces in file name:''
ls *.avi | while read f; do ffmpeg -i "$f" ''...params...'' "$f.mkv"; done
* ''if you're having problems with input format, passing video through mpeg typically helps:''
ffmpeg -i file.mkv -qscale 2 -f mpeg - | ffmpeg -i - ''...params...'' file.avi
* ''downmix 5.1 dolby to normal stereo:''
ffmpeg -i file.mkv ''...params...'' -acodec libmp3lame -ar 44100 -ac 2 file.avi
* ''add an overlay image to the video:''
ffmpeg -i input.mp4 -i image.png -filter_complex "[0:v][1:v] overlay=''x'':''y'':enable='between(t,0,20)'" -pix_fmt yuv420p -c:a copy output.mp4
This will add ''image.png'' as an overlay at position ''x'', ''y'' over the ''imput.mp4'' video, for the first 20 seconds.
* ''fade in and out, first and last 25 frames of a 1000 frame video''
ffmpeg -i input.mp4 -vf "fade=in:0:25,fade=out:975:25" -acodec copy out.mp4
* ''convert album contained in APE or similar file with CUE into separate MP3 tracks''
ffmpeg -i cd.ape -ab 256k cd.mp3
mp3splt -a -c cd.cue cd.mp3

==== [[Short Notes on ffmpeg|more on ffmpeg]] ====

* Rip DVD into FLV (or anything else)
* Video File with Still Image and Audio
* Concatenate Videos

=== [[Short Notes on Flask and Flask-RestPlus|Flask and Flask-RestPlus]] ===

=== [[Short Notes on git|git]] ===

=== [[Short Notes on JS|All things JS - React, Vue, ...]] ===

=== [[Small Kyoto Cabinet and Kyoto Tycoon Notes|Kyoto Cabinet and Kyoto Tycoon]] ===

* Setup and Install Kyoto Cabinet
* Setup and Install Kyoto Tycoon
* KT's RPC API Reference

=== [[Short Notes on Linux Administration|Linux Administration]] ===

* Reset Compiz to Default Settings
* Enable and Start <tt>sendmail</tt>
* Installing <tt>sendmail</tt>
* Sending Emails using <tt>sendmail</tt>
* Find and Mark Bad Blocks
* Using TLP to optimize battery lifetime

=== [[Short Notes on MAC OSX|MAC OSX]] ===

* Asynchronous Execution
** ''NSThread''
** ''performSelectorInBackground''
** ''dispatch_async''
* ''MacPorts''
$ sudo port selfupdate
$ sudo port upgrade outdated
$ sudo port uninstall inactive
$ (just once) sudo port install port_cutleaves
$ sudo port_cutleaves

=== [[Small MongoDB Notes|MongoDB]] ===

* Installation on Ubuntu
* Troubleshooting on Desktop
* Use in PHP via PECL driver

=== [[Short Notes on MySQL|MySQL]] ===

* Error 1045: Access denied
* The <tt>insert ... on duplicate key update</tt> Pitfall

=== [[Short Notes on PDF|PDF]] ===

* Merging PDF Files using GhostScript
* Cutting PDF Files using GhostScript
* Converting to PDF using GhostScript

=== [[Short Notes on PHP|PHP]] ===

* Convert Accented Characters to Non-Accented
* Upload a File using cURL
* HTTP Post in Pure PHP (without cURL)
* Prevent Client Disconnect from Killing PHP
* Disconnect Client and Continue Processing

=== [[Short Notes on Python|Python]] ===

* Timing, and memory, on Linux
* Importing Files
* uWSGI, nginx, Flask
* Decorators
* virtualenv
* SQLAlchemy

=== [[RedBeanPHP Cheat Sheet]] ===

=== [[Short Notes on RESTful APIs|RESTful APIs]] ===

=== [[Short Notes on Security|Security]] ===

* Password Generators
* SSH Access using pubkey's Authentication
* Using RSync together with SSH

=== [[Short Notes on SVN|Subversion/SVN]] ===

* Simple Guide to SVN Client
* Creating a SVN Repository
* Moving a SVN Repository
* Change Repository URL (Subversion 1.7+)
* Setup a SVN Server on a Shared Hosting with no ''root'' Access
* creating and applying patch (the <tt>--diff-cmd</tt> is necessary in case you set some graphical diff-viewer in <tt>~/.subversion/config</tt>):
<pre>cd root_of_project
svn di --diff-cmd diff > ~/patch.diff
cd root_of_other_project
patch -p0 -i ~/patch.diff</pre>
* ''svn:externals'' - pull ''repo1'' into ''dir1'', and ''repo2'' into ''dir2'':
<pre>echo -e 'dir1 svn://repo1\ndir2 svn://repo2' > repos.txt
svn propset svn:externals . -F repos.txt</pre>
validate with " ''svn propget svn:externals .'' " .

=== Ubuntu / Debian / Gnome etc. ===

* restrict gnome app switcher to current workspace only:
<pre>gsettings set org.gnome.shell.app-switcher current-workspace-only true</pre>
* restore or undelete (config) file or directory:
<pre># 1. check what package file or folder belongs to
$ dpkg -S /path/my.conf
# 2.a replace the config file; if it exists, will offer diff
$ sudo apt-get -o Dpkg::Options::="--force-confask" install --reinstall <package-name>
# 2.b restore the directory
$ sudo apt-get -o Dpkg::Options::="--force-confask" install --reinstall $(dpkg -S /etc/some/directory | sed 's/,//g; s/:.*//')</pre>
* increase number of inotify watchers
<pre>$ cat /proc/sys/fs/inotify/max_user_watches
8192
$ echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf
$ sudo sysctl -p
$ cat /proc/sys/fs/inotify/max_user_watches
524288</pre>

==== Up-to-Date Versions of Packages ====

* '''Python'''
Ubuntu 18.04 ships with Python 3.6, but Amazon Linux uses 3.7, so e.g. to use and build Lambda layers and match those against packages you deploy, you need to install 3.7:
<pre>$ sudo add-apt-repository ppa:deadsnakes/ppa
$ sudo apt update
$ sudo apt install python3.7</pre>
Since these do not override default <tt>python</tt> and <tt>python3</tt> links (and better not!), you should create corresponding venv's as:
<pre>$ python3.7 -m venv [env-dir]</pre>

* '''NPM, node.js, etc.'''
The default <tt>apt</tt> ''npm'' version is awfully outdated.

Add NodeSource APT repo; for other versions, e.g. 12.x, change to <tt>setup_12.x</tt>.
<pre>$ curl -sL https://deb.nodesource.com/setup_10.x | sudo -E bash -
$ sudo apt update
$ sudo apt install nodejs</pre>

=== [[Short Notes on ViM|ViM]] ===

* Delete all lines matching a pattern
* Directly type HTML entities
* Reformat a plain text paragraph
<pre>$ vim -x filename - open filename for editing, and save it encrypted (will prompt for a key)
:X - set encryption for current file; use ':set key=' to disable encryption
- put "set cm=blowfish" in your .vimrc to enable safer encryption by default
$ vim +10 [file] ; vim +/hello [file] - open [file] at line 10, or at first occurrence of pattern 'hello' in file
:w !sudo tee % - save file via `sudo` (as root)</pre>

=== [[Short Notes on Wavelets|Wavelets]] ===

* Integer Haar Wavelets in Python

=== [[Short Notes on Web|Web - HTML, CSS, JS, jQuery, etc.]] ===

* Multiple Submit Buttons and/or Multiple Forms on Page
* Make 'float' Elements Appear One under Another
* Upload a File via Form
* Override the IE's Problem with <tt>width</tt>
* Make an Element Fill In All the Remaining Space

=== Other Short Notes ===

==== [[Rip Video DVD that is not Finalized]] ====

==== [[Install Apache and PHP to do Secure h264 Pseudo Streaming]] ====

==== HTTP Response Codes ====

This is not a definitive list, it's just stuff I use the most, and always forget which is which.

* '''100''' - ''Continue''
* '''200''' - ''OK''
* '''201''' - ''Created''
* '''202''' - ''Accepted''
* '''204''' - ''No Content''
* '''301''' - ''Moved Permanently''
* '''302''' - ''Found'' (I typically use this instead of the '''307''' "''Temporary Redirect''")
* '''304''' - ''Not Modified''
* '''400''' - ''Bad Request''
* '''401''' - ''Unauthorized''
* '''403''' - ''Forbidden''
* '''404''' - ''Not Found''
* '''405''' - ''Method Not Allowed'' (using GET on POST calls, etc. - must include "Allow" response header)
* '''406''' - ''Not Acceptable'' (mismatch of response format and "Accept" headers)
* '''500''' - ''Internal Server Error''
* '''501''' - ''Not Implemented''
* '''503''' - ''Service Unavailable''

Main Page

2020-03-20T17:52:16Z

Admin: /* ffmpeg */

Welcome to the '''PaskvilWiki'''!

The future shape of this wiki is unknown for now, let's hope it'll be something good...

This is partially a private wiki - viewing is allowed to all, but editing is allowed only to registered users.

This is more of a prevention than denial - I don't have time to monitor and moderate edits; - you're more than welcome to request registration, all contributors are very much welcome!

Just write me at josef.at.paskvil.dot.com.

== References and Tutorials ==

=== [[emacs and slime reference|Emacs and Slime Reference]] ===

=== [[Small jQuery Reference]] ===

=== [[Small jQuery Cheat Sheet]] ===

=== [[Small SPARQL, RDQL, etc. Cheat Sheet]] ===

== Short Notes - Tips and Tricks ==

=== [[Small 4store How-To|4store]] ===

* setup and installation
* running 4store
* data import
* data import and querying from Python
* Turtle file printer (in Python)

=== [[Short Notes on Apache|Apache]] ===

* Note on serving of local files
* Self-Signed SSL Certificate
* Enable SSL/HTTPS in Apache
* <tt>.htaccess</tt> and <tt>mod_rewrite</tt> Tricks

=== [[Short Notes on AWS|AWS]] ===

* Debugging (lack of) connection to instance
* Authorization header being removed by ElasticBeanstalk
* Notes on IAM
* AWS Lambda with Python - Notes
* boto3 Snippets
* AWS Chalice
* Building a Python Layer

=== [[Short Notes on Bash|Bash]] ===

* Extensions and File Name
* File Test Operators
* ''Commands and Shortcuts''
<pre>$ sudo !! - run last command as root
$ !!:gs/foo/bar - run last command, replacing 'foo' with 'bar'
$ ^foo^bar - run last command, replacing 'foo' with 'bar'
$ cp filename{,.bak} - create a .bak copy of file
<Ctrl-x> <Ctrl-e> - open $EDITOR to edit current command line; upon exit, content will be run
$ <space>command - run the command without saving it in history
$ > file.txt - empty a file
<Alt+.> - put last argument on cursor place; repeat to cycle through
$ (cd /tmp && ls) - jump to /tmp, execute ls, and jump back here
$ pushd /tmp .... $ popd - goes to /tmp, remembering current location; at later time, popd will bring you back
$ \command - run command without using aliases
$ rm !(pattern|pattern) - delete files that do not match given patterns
$ find . -mtime +5 -exec rm {} \; - delete all files in this folder that are older than 5 days
<Ctrl-u> ... <Ctrl-y> - Ctrl+u will cut current command, then you can do whatever you need, and Ctrl+y will paste it back
$ date -d @1234567890 - convert unix timestamp to human-readable</pre>
* ''Informations''
<pre>$ mount | column -t - pretty-print mount table
$ cat /etc/issue - display distro name</pre>
* ''Small Tricks''
<pre>$ echo $(printf '%q' "A\"B") - prints escaped string (in this case, > A\"B <)
$ mount -t tmpfs tmpfs /mnt -o size=1024m - mount temporary RAM partition (fast access, non-persistent!)
$ disown -a && exit - exit the shell while keeping all running processes alive
$ ssh -t remote_host screen -r - directly connect to a remote screen process
$ for I in $(mysql -e 'show databases' -s --skip-column-names); do mysqldump $I | gzip > "$I.sql.gz"; done
- backup all databases into individual files
$ fuser -k filename - kill process that is locking given file
$ readom dev=/dev/scd0 f=/path/to/image.iso - create CD/DVD ISO image
$ diff -rq folder-1 folder-2 - compare 2 folders (files missing or differing)
$ mkdir -p a/deep/dir/tree - create nested directories fast
$ mount file.iso /mnt/iso -o loop - mount ISO file as drive /mnt/iso (dir has to preexist)
$ sudo touch /forcefsck - force a file system check on the next reboot
$ ls | while read f; do process_file "$f"; done - iterate through files even if they contain space in name
$ while read f; do process_file "$f"; done < <(ls) - iterates through files, but does not create sub-process (outside scope is visible)</pre>
* ''lsof''
<pre>$ lsof -nPi - list network connections
$ netstat -tlnp - print all listening ports with PID's (run as root to see all process names)
$ lsof -c gnome-terminal - list files opened by given command
$ lsof +D [folder] - discover all open files in the [folder]
$ lsof -p [pid] - list files opened by given PID
$ lsof -i :8080 - list programs that have port 8080 open ("who to kill" if you get "port already in use")
$ if [ `lsof -- file | wc -l` -ne 0 ] ... - tests if file is open by some process</pre>

=== [[Short Notes on C/C++|C/C++]] ===

* How to clean up after child thread
* Non-blocking IO using sockets
* Shared Libraries on Linux+GCC
* Singleton
* <tt>std::move</tt> Semantics
* Converting between <tt>std::string</tt> and <tt>std::wstring</tt> in C++11

=== cURL (in terminal) ===

Use '''-s''' option to make <tt>curl</tt> silent.

* POST to server using cURL:
<pre>curl --data "key1=value1&key2=value2" http://www.example.com/post.php</pre>
* Upload to FTP server (with credentials), having connection timeout 60 seconds, and 120 seconds maximum operation time:
<pre>curl --connect-timeout 60 -m 120 -T file.ext "ftp://12.34.56.78/folder" --user user:pass</pre>
* Download from FTP server (with credentials):
<pre>curl --user user:pass "ftp://12.34.56.78/folder/file.ext" -o ~/Downloads/file.ext</pre>

=== [[Small Docker Notes|Docker]] ===

=== [[Small elasticsearch Notes|ElasticSearch]] ===

* Installation
* What You Get
* Indexing, Routing, etc.
* Deleting
* Getting and Multi-Getting
* Updating
* Search and Query DSL

=== [[Short Notes on ffmpeg|ffmpeg]] ===

* ''process all files, may include spaces in file name:''
ls *.avi | while read f; do ffmpeg -i "$f" ''...params...'' "$f.mkv"; done
* ''if you're having problems with input format, passing video through mpeg typically helps:''
ffmpeg -i file.mkv -qscale 2 -f mpeg - | ffmpeg -i - ''...params...'' file.avi
* ''downmix 5.1 dolby to normal stereo:''
ffmpeg -i file.mkv ''...params...'' -acodec libmp3lame -ar 44100 -ac 2 file.avi
* ''add an overlay image to the video:''
ffmpeg -i input.mp4 -i image.png -filter_complex "[0:v][1:v] overlay=''x'':''y'':enable='between(t,0,20)'" -pix_fmt yuv420p -c:a copy output.mp4
This will add ''image.png'' as an overlay at position ''x'', ''y'' over the ''imput.mp4'' video, for the first 20 seconds.
* ''fade in and out, first and last 25 frames of a 1000 frame video''
ffmpeg -i input.mp4 -vf "fade=in:0:25,fade=out:975:25" -acodec copy out.mp4
* ''convert album contained in APE or similar file with CUE into separate MP3 tracks''
ffmpeg -i cd.ape -ab 256k cd.mp3
mp3splt -a -c cd.cue cd.mp3

==== [[Short Notes on ffmpeg|more on ffmpeg]] ====

* Rip DVD into FLV (or anything else)
* Video File with Still Image and Audio
* Concatenate Videos

=== [[Short Notes on Flask and Flask-RestPlus|Flask and Flask-RestPlus]] ===

=== [[Short Notes on git|git]] ===

=== [[Short Notes on JS|All things JS - React, Vue, ...]] ===

=== [[Small Kyoto Cabinet and Kyoto Tycoon Notes|Kyoto Cabinet and Kyoto Tycoon]] ===

* Setup and Install Kyoto Cabinet
* Setup and Install Kyoto Tycoon
* KT's RPC API Reference

=== [[Short Notes on Linux Administration|Linux Administration]] ===

* Reset Compiz to Default Settings
* Enable and Start <tt>sendmail</tt>
* Installing <tt>sendmail</tt>
* Sending Emails using <tt>sendmail</tt>
* Find and Mark Bad Blocks
* Using TLP to optimize battery lifetime

=== [[Short Notes on MAC OSX|MAC OSX]] ===

* Asynchronous Execution
** ''NSThread''
** ''performSelectorInBackground''
** ''dispatch_async''
* ''MacPorts''
$ sudo port selfupdate
$ sudo port upgrade outdated
$ sudo port uninstall inactive
$ (just once) sudo port install port_cutleaves
$ sudo port_cutleaves

=== [[Small MongoDB Notes|MongoDB]] ===

* Installation on Ubuntu
* Troubleshooting on Desktop
* Use in PHP via PECL driver

=== [[Short Notes on MySQL|MySQL]] ===

* Error 1045: Access denied
* The <tt>insert ... on duplicate key update</tt> Pitfall

=== [[Short Notes on PDF|PDF]] ===

* Merging PDF Files using GhostScript
* Cutting PDF Files using GhostScript
* Converting to PDF using GhostScript

=== [[Short Notes on PHP|PHP]] ===

* Convert Accented Characters to Non-Accented
* Upload a File using cURL
* HTTP Post in Pure PHP (without cURL)
* Prevent Client Disconnect from Killing PHP
* Disconnect Client and Continue Processing

=== [[Short Notes on Python|Python]] ===

* Timing, and memory, on Linux
* Importing Files
* uWSGI, nginx, Flask
* Decorators
* virtualenv
* SQLAlchemy

=== [[RedBeanPHP Cheat Sheet]] ===

=== [[Short Notes on RESTful APIs|RESTful APIs]] ===

=== [[Short Notes on Security|Security]] ===

* Password Generators
* SSH Access using pubkey's Authentication
* Using RSync together with SSH

=== [[Short Notes on SVN|Subversion/SVN]] ===

* Simple Guide to SVN Client
* Creating a SVN Repository
* Moving a SVN Repository
* Change Repository URL (Subversion 1.7+)
* Setup a SVN Server on a Shared Hosting with no ''root'' Access
* creating and applying patch (the <tt>--diff-cmd</tt> is necessary in case you set some graphical diff-viewer in <tt>~/.subversion/config</tt>):
<pre>cd root_of_project
svn di --diff-cmd diff > ~/patch.diff
cd root_of_other_project
patch -p0 -i ~/patch.diff</pre>
* ''svn:externals'' - pull ''repo1'' into ''dir1'', and ''repo2'' into ''dir2'':
<pre>echo -e 'dir1 svn://repo1\ndir2 svn://repo2' > repos.txt
svn propset svn:externals . -F repos.txt</pre>
validate with " ''svn propget svn:externals .'' " .

=== Ubuntu / Debian / Gnome etc. ===

* restrict gnome app switcher to current workspace only:
<pre>gsettings set org.gnome.shell.app-switcher current-workspace-only true</pre>
* restore or undelete (config) file or directory:
<pre># 1. check what package file or folder belongs to
$ dpkg -S /path/my.conf
# 2.a replace the config file; if it exists, will offer diff
$ sudo apt-get -o Dpkg::Options::="--force-confask" install --reinstall <package-name>
# 2.b restore the directory
$ sudo apt-get -o Dpkg::Options::="--force-confask" install --reinstall $(dpkg -S /etc/some/directory | sed 's/,//g; s/:.*//')</pre>
* increase number of inotify watchers
<pre>$ cat /proc/sys/fs/inotify/max_user_watches
8192
$ echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf
$ sudo sysctl -p
$ cat /proc/sys/fs/inotify/max_user_watches
524288</pre>

==== Up-to-Date Versions of Packages ====

* '''Python'''
Ubuntu 18.04 ships with Python 3.6, but Amazon Linux uses 3.7, so e.g. to use and build Lambda layers and match those against packages you deploy, you need to install 3.7:
<pre>$ sudo add-apt-repository ppa:deadsnakes/ppa
$ sudo apt update
$ sudo apt install python3.7</pre>
Since these do not override default <tt>python</tt> and <tt>python3</tt> links (and better not!), you should create corresponding venv's as:
<pre>$ python3.7 -m venv [env-dir]</pre>

* '''NPM, node.js, etc.'''
The default <tt>apt</tt> ''npm'' version is awfully outdated.

Add NodeSource APT repo; for other versions, e.g. 12.x, change to <tt>setup_12.x</tt>.
<pre>$ curl -sL https://deb.nodesource.com/setup_10.x | sudo -E bash -
$ sudo apt update
$ sudo apt install nodejs</pre>

=== [[Short Notes on ViM|ViM]] ===

* Delete all lines matching a pattern
* Directly type HTML entities
* Reformat a plain text paragraph
<pre>$ vim -x filename - open filename for editing, and save it encrypted (will prompt for a key)
:X - set encryption for current file; use ':set key=' to disable encryption
- put "set cm=blowfish" in your .vimrc to enable safer encryption by default
$ vim +10 [file] ; vim +/hello [file] - open [file] at line 10, or at first occurrence of pattern 'hello' in file
:w !sudo tee % - save file via `sudo` (as root)</pre>

=== [[Short Notes on Wavelets|Wavelets]] ===

* Integer Haar Wavelets in Python

=== [[Short Notes on Web|Web - HTML, CSS, JS, jQuery, etc.]] ===

* Multiple Submit Buttons and/or Multiple Forms on Page
* Make 'float' Elements Appear One under Another
* Upload a File via Form
* Override the IE's Problem with <tt>width</tt>
* Make an Element Fill In All the Remaining Space

=== Other Short Notes ===

==== [[Rip Video DVD that is not Finalized]] ====

==== [[Install Apache and PHP to do Secure h264 Pseudo Streaming]] ====

==== HTTP Response Codes ====

This is not a definitive list, it's just stuff I use the most, and always forget which is which.

* '''100''' - ''Continue''
* '''200''' - ''OK''
* '''201''' - ''Created''
* '''202''' - ''Accepted''
* '''204''' - ''No Content''
* '''301''' - ''Moved Permanently''
* '''302''' - ''Found'' (I typically use this instead of the '''307''' "''Temporary Redirect''")
* '''304''' - ''Not Modified''
* '''400''' - ''Bad Request''
* '''401''' - ''Unauthorized''
* '''403''' - ''Forbidden''
* '''404''' - ''Not Found''
* '''405''' - ''Method Not Allowed'' (using GET on POST calls, etc. - must include "Allow" response header)
* '''406''' - ''Not Acceptable'' (mismatch of response format and "Accept" headers)
* '''500''' - ''Internal Server Error''
* '''501''' - ''Not Implemented''
* '''503''' - ''Service Unavailable''

Short Notes on Python

2020-02-28T07:22:03Z

Admin: /* Importing Files */

== Timing, and memory, on Linux ==

=== Timing ===

On Linux, it's safer to use <tt>time.time()</tt>
<pre>import time
t = time.time()
# do some stuff
print "stuff took %1.3f", time.time() - t, "seconds"</pre>
On Windows, AFAIK, it's safer to use <tt>time.clock()</tt>

=== Memory ===

For me, the following does a good job getting memory usage (in kB) on Linux:
<pre>import resource
print resource.getrusage(resource.RUSAGE_SELF).ru_maxrss</pre>
Since <tt>resource</tt> is standard package, it should work on Windows too, but I don't know if it does, or what units are used if it works.

== Importing Files ==

If you need to import a file '../mylib/commons.py', you can use the following snippet:
<nowiki>import sys, os
fld = os.path.realpath(os.path.abspath(os.path.join('..', 'mylib')))
if fld not in sys.path:
sys.path.insert(0, fld)
import commons

# use your commons.py module now...
commons.super_function()</nowiki>

== Merging Dictionaries ==

Below are few solutions, the first one works for any list of dictionaries, the rest is just for 2 dict's, with the last 2 suitable for inlining:

<pre># as a generic function that merges list of dict's
def merge_dicts(dicts: list) -> dict:
res = dict()
for d in dicts:
res.update(d)
return res

# if you're merging known number of dict's:

def merge_two_1(a: dict, b: dict) -> dict:
return dict(a, **b)

# python 3.5+, the fastest of the lot
def merge_two_1(a: dict, b: dict) -> dict:
return {**a, **b}</pre>

== uWSGI, nginx, Flask ==

* install uwsgi (incl. uwsgi python plugin), python flask, and nginx,

=== Setting Up uWSGI ===

* create ''main.py'' file that will hold the server logic, for instance:
<pre>from flask import Flask

app = Flask(__name__)

@app.route("/")
def hello():
return "hello there!"</pre>
* create uwsgi config file, ''wsgi.ini'' ('''minimal''' version here; read uwsgi docs for head-spinning array of configurables):
<pre>[uwsgi]
module = main:app
master = true
processes = 5
socket = 127.0.0.1:8000
protocol = http
plugin = python</pre>
* run uwsgi
uwsgi --ini wsgi.ini
* check that all works on http://localhost:8000/

=== Adding nginx Layer ===

* remove the "protocol" directive from ''wsgi.ini'', and add "die-on-term":
<pre>[uwsgi]
module = main:app
master = true
processes = 5
socket = 127.0.0.1:8000
plugin = python
die-on-term = true</pre>
* add a new vhost to nginx - ''/etc/nginx/sites-available/app.nginx'':
<pre>server {
listen 80;
server_name my.awesome.domain;
location / {
include uwsgi_params;
uwsgi_pass 127.0.0.1:8000;
}
}</pre>
** communication through socket is also possible (see ''socket'', ''chmod-socket'', ''vacuum'' and other directives for uWSGI)
** of course, create link in ''/etc/nginx/sites-enabled/'', and restart nginx,

=== Run uWSGI daemon on boot - supervisor ===

* install <tt>supervisor</tt>
apt-get install supervisor
* add/edit <tt>/etc/supervisord.conf</tt> with content like this:
<pre>[unix_http_server]
file=/run/supervisor.sock

[supervisord]
logfile=/var/log/supervisord.log
logfile_maxbytes=1MB
logfile_backups=1
loglevel=info ; (others: warn,debug,trace)
pidfile=/run/supervisord.pid
nodaemon=false
minfds=1024
minprocs=200

[supervisorctl]
serverurl=unix:///run/supervisor.sock

[program:your app]
directory=/path/to/app
command=/usr/bin/uwsgi --ini wsgi.ini
autostart=true
autorestart=true
stdout_logfile=/var/log/uwsgi.log
redirect_stderr=true
stopsignal=QUIT</pre>

=== Run uWSGI daemon on boot - systemd ===

* create ''systemd'' file for uWSGI, ''/etc/systemd/system/uwsgi-app.service'':
<pre>[Unit]
Description=Job that runs the uWSGI app

[Service]
Type=simple
WorkingDirectory=/home/project/flask-test/
ExecStart=/usr/bin/uwsgi --ini wsgi.ini

[Install]
WantedBy=multi-user.target</pre>

Then you can start and stop the uwsgi service using:
# systemctl start uwsgi-app.service
# systemctl stop uwsgi-app.service

Once you're happy with the settings, ''enable'' the daemon to be run on boot:
# systemctl enable uwsgi-app.service

== Decorators ==

''needs a bit of refreshment and updating...''

[http://www.artima.com/weblogs/viewpost.jsp?thread=240845 based on].

Decorators are simple and expressive way to modify function without editing the source of the function itself. Or, the other way around, to modify multiple functions in the same way, without code duplication (DRY).

Decorators can be spotted in the code by starting with <tt>@</tt> character. Decorator is a function (or class) that can do some additional work before or after the call to the decorated function. It can even call the decorated function multiple times, or not at all.

Decorators can be implemented as closures (my fave), or as classes; the following approaches are equivalent:

<pre># using a function (closure) as a decorator
def beforeAndAfter(f):
def decorated_fn():
print("Before", f.__name__)
f()
print("After", f.__name__)
return decorated_fn

@entryExit
def func():
print "func() is in da' house!"

func()</pre>

<pre># using a class as a decorator
class beforeAndAfter(object):

def __init__(self, f):
self.f = f

def __call__(self):
print("Before", self.f.__name__)
self.f()
print("After", self.f.__name__)

@beforeAndAfter
def func():
print "func() is in da' house!"

func()</pre>

Note that in functional decorator, anything outside the body of <tt>decorated_fn()</tt> is equivalent to content of the <tt>__init__()</tt> constructor of the class-based decorator. This code is run during the initialization, only once for each decorated function, regardless of whether the decorated function is ever called in the code - you should avoid any heavy lifting there.

Decorators can also accept arguments, and (obviously should) forward arguments to the decorated function:

<pre>def beforeAndAfter(p1, p2):
def wrap(f):
def wrapped_f(*args, **kwargs):
print "Decorator arguments:", p1, p2
print("Before", f.__name__)
f(*args, **kwargs)
print("After", f.__name__)
return wrapped_f
return wrap

@beforeAndAfter("hello", "world")
def func(a, b=2):
print "func() is in da' house,", a, b

func(1)</pre>

== virtualenv ==

I prefer installing <tt>virtualenv</tt> tool through <tt>pip</tt>, to make sure those are in sync version-wise.

<pre># create a new venv
# it is better to keep venv's in some separate folder, not to pollute your project folder
$ virtualenv ~/.virtualenvs/my-new-env

# "log into" your venv; success can be seen by your command line being prefixed by venv name
$ source ~/.virtualenvs/my-new-env/bin/activate

# now you can install anything you need, tucked away in your venv:
(my-new-env) $ pip install -r requirements.txt

# when done, just deactivate
(my-new-env) $ deactivate</pre>

== SQLAlchemy ==

=== Make SQLAlchemy Read-Only ===

All write operations in SQLAlchemy pass through <tt>flush()</tt> method of your session.
Just monkey-path it to do nothing!
<pre>engine = create_engine("connection string")
Session = sessionmaker(bind=engine, autoflush=False, autocommit=False)
session = Session()
session.flush = lambda *args,**kwargs: None</pre>

=== Making Session Self-Recoverable ===

Some DB's, most prominently Postgres, do not recover well from errors (e.g. Pg just keeps saying that there's error in current transaction).
One way is to rollback the transaction on "any" error:

<pre>from contextlib import contextmanager

@contextmanager
def db_session():
session = init() # get the session any way you like
try:
yield session
except:
session.rollback()
raise
else:
session.commit()

def use_db():
with db_session() as session:
# use session object</pre>

=== Show Create Statement for Your Model ===

<pre>from sqlalchemy.schema import CreateTable

with db_session():
print(CreateTable(MyModel.__table__))</pre>

or, without actually instantiating an engine:

<pre>from sqlalchemy.dialects import postgresql

print(CreateTable(MyModel.__table__).compile(dialect=postgresql.dialect()))</pre>

=== List Tables and Their Columns ===

<pre>import sqlalchemy as sqla

db_conn_str = "..your connection string.."
engine = sqla.create_engine(db_conn_str)
inspector = sqla.inspect(engine)
schemas = inspector.get_schema_names()

for schema in schemas:
print("schema: {}".format(schema))
for table_name in inspector.get_table_names(schema=schema):
cols = ['{} ({})'.format(col['name'], col['type']) for col in inspector.get_columns(table_name, schema=schema)]
print("Table: {} ({})".format(table_name, ', '.join(cols)))</pre>

Short Notes on Python

2020-02-14T19:29:19Z

Admin: /* Show Create Statement for Your Model */

== Timing, and memory, on Linux ==

=== Timing ===

On Linux, it's safer to use <tt>time.time()</tt>
<pre>import time
t = time.time()
# do some stuff
print "stuff took %1.3f", time.time() - t, "seconds"</pre>
On Windows, AFAIK, it's safer to use <tt>time.clock()</tt>

=== Memory ===

For me, the following does a good job getting memory usage (in kB) on Linux:
<pre>import resource
print resource.getrusage(resource.RUSAGE_SELF).ru_maxrss</pre>
Since <tt>resource</tt> is standard package, it should work on Windows too, but I don't know if it does, or what units are used if it works.

== Importing Files ==

If you need to import a file '../mylib/commons.py', you can use the following snippet:
<nowiki>import sys, os
fld = os.path.realpath(os.path.abspath(os.path.join('..', 'mylib')))
if fld not in sys.path:
sys.path.insert(0, fld)
import commons

# use your commons.py module now...
commons.super_function()</nowiki>

== uWSGI, nginx, Flask ==

* install uwsgi (incl. uwsgi python plugin), python flask, and nginx,

=== Setting Up uWSGI ===

* create ''main.py'' file that will hold the server logic, for instance:
<pre>from flask import Flask

app = Flask(__name__)

@app.route("/")
def hello():
return "hello there!"</pre>
* create uwsgi config file, ''wsgi.ini'' ('''minimal''' version here; read uwsgi docs for head-spinning array of configurables):
<pre>[uwsgi]
module = main:app
master = true
processes = 5
socket = 127.0.0.1:8000
protocol = http
plugin = python</pre>
* run uwsgi
uwsgi --ini wsgi.ini
* check that all works on http://localhost:8000/

=== Adding nginx Layer ===

* remove the "protocol" directive from ''wsgi.ini'', and add "die-on-term":
<pre>[uwsgi]
module = main:app
master = true
processes = 5
socket = 127.0.0.1:8000
plugin = python
die-on-term = true</pre>
* add a new vhost to nginx - ''/etc/nginx/sites-available/app.nginx'':
<pre>server {
listen 80;
server_name my.awesome.domain;
location / {
include uwsgi_params;
uwsgi_pass 127.0.0.1:8000;
}
}</pre>
** communication through socket is also possible (see ''socket'', ''chmod-socket'', ''vacuum'' and other directives for uWSGI)
** of course, create link in ''/etc/nginx/sites-enabled/'', and restart nginx,

=== Run uWSGI daemon on boot - supervisor ===

* install <tt>supervisor</tt>
apt-get install supervisor
* add/edit <tt>/etc/supervisord.conf</tt> with content like this:
<pre>[unix_http_server]
file=/run/supervisor.sock

[supervisord]
logfile=/var/log/supervisord.log
logfile_maxbytes=1MB
logfile_backups=1
loglevel=info ; (others: warn,debug,trace)
pidfile=/run/supervisord.pid
nodaemon=false
minfds=1024
minprocs=200

[supervisorctl]
serverurl=unix:///run/supervisor.sock

[program:your app]
directory=/path/to/app
command=/usr/bin/uwsgi --ini wsgi.ini
autostart=true
autorestart=true
stdout_logfile=/var/log/uwsgi.log
redirect_stderr=true
stopsignal=QUIT</pre>

=== Run uWSGI daemon on boot - systemd ===

* create ''systemd'' file for uWSGI, ''/etc/systemd/system/uwsgi-app.service'':
<pre>[Unit]
Description=Job that runs the uWSGI app

[Service]
Type=simple
WorkingDirectory=/home/project/flask-test/
ExecStart=/usr/bin/uwsgi --ini wsgi.ini

[Install]
WantedBy=multi-user.target</pre>

Then you can start and stop the uwsgi service using:
# systemctl start uwsgi-app.service
# systemctl stop uwsgi-app.service

Once you're happy with the settings, ''enable'' the daemon to be run on boot:
# systemctl enable uwsgi-app.service

== Decorators ==

''needs a bit of refreshment and updating...''

[http://www.artima.com/weblogs/viewpost.jsp?thread=240845 based on].

Decorators are simple and expressive way to modify function without editing the source of the function itself. Or, the other way around, to modify multiple functions in the same way, without code duplication (DRY).

Decorators can be spotted in the code by starting with <tt>@</tt> character. Decorator is a function (or class) that can do some additional work before or after the call to the decorated function. It can even call the decorated function multiple times, or not at all.

Decorators can be implemented as closures (my fave), or as classes; the following approaches are equivalent:

<pre># using a function (closure) as a decorator
def beforeAndAfter(f):
def decorated_fn():
print("Before", f.__name__)
f()
print("After", f.__name__)
return decorated_fn

@entryExit
def func():
print "func() is in da' house!"

func()</pre>

<pre># using a class as a decorator
class beforeAndAfter(object):

def __init__(self, f):
self.f = f

def __call__(self):
print("Before", self.f.__name__)
self.f()
print("After", self.f.__name__)

@beforeAndAfter
def func():
print "func() is in da' house!"

func()</pre>

Note that in functional decorator, anything outside the body of <tt>decorated_fn()</tt> is equivalent to content of the <tt>__init__()</tt> constructor of the class-based decorator. This code is run during the initialization, only once for each decorated function, regardless of whether the decorated function is ever called in the code - you should avoid any heavy lifting there.

Decorators can also accept arguments, and (obviously should) forward arguments to the decorated function:

<pre>def beforeAndAfter(p1, p2):
def wrap(f):
def wrapped_f(*args, **kwargs):
print "Decorator arguments:", p1, p2
print("Before", f.__name__)
f(*args, **kwargs)
print("After", f.__name__)
return wrapped_f
return wrap

@beforeAndAfter("hello", "world")
def func(a, b=2):
print "func() is in da' house,", a, b

func(1)</pre>

== virtualenv ==

I prefer installing <tt>virtualenv</tt> tool through <tt>pip</tt>, to make sure those are in sync version-wise.

<pre># create a new venv
# it is better to keep venv's in some separate folder, not to pollute your project folder
$ virtualenv ~/.virtualenvs/my-new-env

# "log into" your venv; success can be seen by your command line being prefixed by venv name
$ source ~/.virtualenvs/my-new-env/bin/activate

# now you can install anything you need, tucked away in your venv:
(my-new-env) $ pip install -r requirements.txt

# when done, just deactivate
(my-new-env) $ deactivate</pre>

== SQLAlchemy ==

=== Make SQLAlchemy Read-Only ===

All write operations in SQLAlchemy pass through <tt>flush()</tt> method of your session.
Just monkey-path it to do nothing!
<pre>engine = create_engine("connection string")
Session = sessionmaker(bind=engine, autoflush=False, autocommit=False)
session = Session()
session.flush = lambda *args,**kwargs: None</pre>

=== Making Session Self-Recoverable ===

Some DB's, most prominently Postgres, do not recover well from errors (e.g. Pg just keeps saying that there's error in current transaction).
One way is to rollback the transaction on "any" error:

<pre>from contextlib import contextmanager

@contextmanager
def db_session():
session = init() # get the session any way you like
try:
yield session
except:
session.rollback()
raise
else:
session.commit()

def use_db():
with db_session() as session:
# use session object</pre>

=== Show Create Statement for Your Model ===

<pre>from sqlalchemy.schema import CreateTable

with db_session():
print(CreateTable(MyModel.__table__))</pre>

or, without actually instantiating an engine:

<pre>from sqlalchemy.dialects import postgresql

print(CreateTable(MyModel.__table__).compile(dialect=postgresql.dialect()))</pre>

=== List Tables and Their Columns ===

<pre>import sqlalchemy as sqla

db_conn_str = "..your connection string.."
engine = sqla.create_engine(db_conn_str)
inspector = sqla.inspect(engine)
schemas = inspector.get_schema_names()

for schema in schemas:
print("schema: {}".format(schema))
for table_name in inspector.get_table_names(schema=schema):
cols = ['{} ({})'.format(col['name'], col['type']) for col in inspector.get_columns(table_name, schema=schema)]
print("Table: {} ({})".format(table_name, ', '.join(cols)))</pre>

Short Notes on Python

2020-02-14T14:14:09Z

Admin: /* SQLAlchemy */

== Timing, and memory, on Linux ==

=== Timing ===

On Linux, it's safer to use <tt>time.time()</tt>
<pre>import time
t = time.time()
# do some stuff
print "stuff took %1.3f", time.time() - t, "seconds"</pre>
On Windows, AFAIK, it's safer to use <tt>time.clock()</tt>

=== Memory ===

For me, the following does a good job getting memory usage (in kB) on Linux:
<pre>import resource
print resource.getrusage(resource.RUSAGE_SELF).ru_maxrss</pre>
Since <tt>resource</tt> is standard package, it should work on Windows too, but I don't know if it does, or what units are used if it works.

== Importing Files ==

If you need to import a file '../mylib/commons.py', you can use the following snippet:
<nowiki>import sys, os
fld = os.path.realpath(os.path.abspath(os.path.join('..', 'mylib')))
if fld not in sys.path:
sys.path.insert(0, fld)
import commons

# use your commons.py module now...
commons.super_function()</nowiki>

== uWSGI, nginx, Flask ==

* install uwsgi (incl. uwsgi python plugin), python flask, and nginx,

=== Setting Up uWSGI ===

* create ''main.py'' file that will hold the server logic, for instance:
<pre>from flask import Flask

app = Flask(__name__)

@app.route("/")
def hello():
return "hello there!"</pre>
* create uwsgi config file, ''wsgi.ini'' ('''minimal''' version here; read uwsgi docs for head-spinning array of configurables):
<pre>[uwsgi]
module = main:app
master = true
processes = 5
socket = 127.0.0.1:8000
protocol = http
plugin = python</pre>
* run uwsgi
uwsgi --ini wsgi.ini
* check that all works on http://localhost:8000/

=== Adding nginx Layer ===

* remove the "protocol" directive from ''wsgi.ini'', and add "die-on-term":
<pre>[uwsgi]
module = main:app
master = true
processes = 5
socket = 127.0.0.1:8000
plugin = python
die-on-term = true</pre>
* add a new vhost to nginx - ''/etc/nginx/sites-available/app.nginx'':
<pre>server {
listen 80;
server_name my.awesome.domain;
location / {
include uwsgi_params;
uwsgi_pass 127.0.0.1:8000;
}
}</pre>
** communication through socket is also possible (see ''socket'', ''chmod-socket'', ''vacuum'' and other directives for uWSGI)
** of course, create link in ''/etc/nginx/sites-enabled/'', and restart nginx,

=== Run uWSGI daemon on boot - supervisor ===

* install <tt>supervisor</tt>
apt-get install supervisor
* add/edit <tt>/etc/supervisord.conf</tt> with content like this:
<pre>[unix_http_server]
file=/run/supervisor.sock

[supervisord]
logfile=/var/log/supervisord.log
logfile_maxbytes=1MB
logfile_backups=1
loglevel=info ; (others: warn,debug,trace)
pidfile=/run/supervisord.pid
nodaemon=false
minfds=1024
minprocs=200

[supervisorctl]
serverurl=unix:///run/supervisor.sock

[program:your app]
directory=/path/to/app
command=/usr/bin/uwsgi --ini wsgi.ini
autostart=true
autorestart=true
stdout_logfile=/var/log/uwsgi.log
redirect_stderr=true
stopsignal=QUIT</pre>

=== Run uWSGI daemon on boot - systemd ===

* create ''systemd'' file for uWSGI, ''/etc/systemd/system/uwsgi-app.service'':
<pre>[Unit]
Description=Job that runs the uWSGI app

[Service]
Type=simple
WorkingDirectory=/home/project/flask-test/
ExecStart=/usr/bin/uwsgi --ini wsgi.ini

[Install]
WantedBy=multi-user.target</pre>

Then you can start and stop the uwsgi service using:
# systemctl start uwsgi-app.service
# systemctl stop uwsgi-app.service

Once you're happy with the settings, ''enable'' the daemon to be run on boot:
# systemctl enable uwsgi-app.service

== Decorators ==

''needs a bit of refreshment and updating...''

[http://www.artima.com/weblogs/viewpost.jsp?thread=240845 based on].

Decorators are simple and expressive way to modify function without editing the source of the function itself. Or, the other way around, to modify multiple functions in the same way, without code duplication (DRY).

Decorators can be spotted in the code by starting with <tt>@</tt> character. Decorator is a function (or class) that can do some additional work before or after the call to the decorated function. It can even call the decorated function multiple times, or not at all.

Decorators can be implemented as closures (my fave), or as classes; the following approaches are equivalent:

<pre># using a function (closure) as a decorator
def beforeAndAfter(f):
def decorated_fn():
print("Before", f.__name__)
f()
print("After", f.__name__)
return decorated_fn

@entryExit
def func():
print "func() is in da' house!"

func()</pre>

<pre># using a class as a decorator
class beforeAndAfter(object):

def __init__(self, f):
self.f = f

def __call__(self):
print("Before", self.f.__name__)
self.f()
print("After", self.f.__name__)

@beforeAndAfter
def func():
print "func() is in da' house!"

func()</pre>

Note that in functional decorator, anything outside the body of <tt>decorated_fn()</tt> is equivalent to content of the <tt>__init__()</tt> constructor of the class-based decorator. This code is run during the initialization, only once for each decorated function, regardless of whether the decorated function is ever called in the code - you should avoid any heavy lifting there.

Decorators can also accept arguments, and (obviously should) forward arguments to the decorated function:

<pre>def beforeAndAfter(p1, p2):
def wrap(f):
def wrapped_f(*args, **kwargs):
print "Decorator arguments:", p1, p2
print("Before", f.__name__)
f(*args, **kwargs)
print("After", f.__name__)
return wrapped_f
return wrap

@beforeAndAfter("hello", "world")
def func(a, b=2):
print "func() is in da' house,", a, b

func(1)</pre>

== virtualenv ==

I prefer installing <tt>virtualenv</tt> tool through <tt>pip</tt>, to make sure those are in sync version-wise.

<pre># create a new venv
# it is better to keep venv's in some separate folder, not to pollute your project folder
$ virtualenv ~/.virtualenvs/my-new-env

# "log into" your venv; success can be seen by your command line being prefixed by venv name
$ source ~/.virtualenvs/my-new-env/bin/activate

# now you can install anything you need, tucked away in your venv:
(my-new-env) $ pip install -r requirements.txt

# when done, just deactivate
(my-new-env) $ deactivate</pre>

== SQLAlchemy ==

=== Make SQLAlchemy Read-Only ===

All write operations in SQLAlchemy pass through <tt>flush()</tt> method of your session.
Just monkey-path it to do nothing!
<pre>engine = create_engine("connection string")
Session = sessionmaker(bind=engine, autoflush=False, autocommit=False)
session = Session()
session.flush = lambda *args,**kwargs: None</pre>

=== Making Session Self-Recoverable ===

Some DB's, most prominently Postgres, do not recover well from errors (e.g. Pg just keeps saying that there's error in current transaction).
One way is to rollback the transaction on "any" error:

<pre>from contextlib import contextmanager

@contextmanager
def db_session():
session = init() # get the session any way you like
try:
yield session
except:
session.rollback()
raise
else:
session.commit()

def use_db():
with db_session() as session:
# use session object</pre>

=== Show Create Statement for Your Model ===

<pre>from sqlalchemy.schema import CreateTable
with db_session():
print(CreateTable(MyModel.__table__))</pre>

=== List Tables and Their Columns ===

<pre>import sqlalchemy as sqla

db_conn_str = "..your connection string.."
engine = sqla.create_engine(db_conn_str)
inspector = sqla.inspect(engine)
schemas = inspector.get_schema_names()

for schema in schemas:
print("schema: {}".format(schema))
for table_name in inspector.get_table_names(schema=schema):
cols = ['{} ({})'.format(col['name'], col['type']) for col in inspector.get_columns(table_name, schema=schema)]
print("Table: {} ({})".format(table_name, ', '.join(cols)))</pre>

Short Notes on JS

2020-02-04T15:14:11Z

Admin:

== Fetch Patterns ==

<pre>fetch("url", { options... })
.then((response) => {
if (!response.ok)
throw new Error('Network response was not ok');
return response.json(); // or response.blob(), etc.
})
.then((data) => {
// do something with the data received
})
.catch((error) => {
console.error('Failed to fetch:', error);
});</pre>

== Environment Variables in WebPack ==

WebPack does not have access to environment (duh), so you need to "bake" any relevant environment variables in the WebPack during build:
<pre>new webpack.DefinePlugin({
'process.env': {
NODE_ENV: JSON.stringify(process.env.NODE_ENV),
STAGE: JSON.stringify(process.env.STAGE),
// ...
}
})</pre>

== Debugging in VS Code on Linux ==

* install ''Debugger for Chrome'' extension
* open the ''launch.json'' file, and edit as follows:
** custom profile - ''user-data-dir'' - is used, to make sure you have clean (enough) slate
<pre>{
// Use IntelliSense to learn about possible attributes.
// Hover to view descriptions of existing attributes.
// For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
"version": "0.2.0",
"configurations": [
{
"type": "chrome",
"request": "launch",
"runtimeExecutable": "/usr/bin/chromium-browser",
"runtimeArgs": ["--remote-debugging-port=9222", "--user-data-dir=/home/user/tmp/remote-profile"],
"name": "Launch Chrome against localhost",
"url": "http://localhost:3000",
"webRoot": "${workspaceFolder}"
}
]
}</pre>
* run the npm (the port used should correspond to the ''url'' key above)
* hit debug in VS Code, add breakpoints, etc., and enjoy!
** this will actually open new browser window, as subwindow of VS Code, neat!

Short Notes on JS

2020-01-28T11:50:57Z

Admin:

2019-11-21T08:12:58Z

Admin: /* Linux Administration */

Welcome to the '''PaskvilWiki'''!

The future shape of this wiki is unknown for now, let's hope it'll be something good...

This is partially a private wiki - viewing is allowed to all, but editing is allowed only to registered users.

This is more of a prevention than denial - I don't have time to monitor and moderate edits; - you're more than welcome to request registration, all contributors are very much welcome!

Just write me at josef.at.paskvil.dot.com.

== References and Tutorials ==

=== [[emacs and slime reference|Emacs and Slime Reference]] ===

=== [[Small jQuery Reference]] ===

=== [[Small jQuery Cheat Sheet]] ===

=== [[Small SPARQL, RDQL, etc. Cheat Sheet]] ===

== Short Notes - Tips and Tricks ==

=== [[Small 4store How-To|4store]] ===

* setup and installation
* running 4store
* data import
* data import and querying from Python
* Turtle file printer (in Python)

=== [[Short Notes on Apache|Apache]] ===

* Note on serving of local files
* Self-Signed SSL Certificate
* Enable SSL/HTTPS in Apache
* <tt>.htaccess</tt> and <tt>mod_rewrite</tt> Tricks

=== [[Short Notes on AWS|AWS]] ===

* Debugging (lack of) connection to instance
* Authorization header being removed by ElasticBeanstalk

=== [[Short Notes on Bash|Bash]] ===

* Extensions and File Name
* File Test Operators
* ''Commands and Shortcuts''
<pre>$ sudo !! - run last command as root
$ !!:gs/foo/bar - run last command, replacing 'foo' with 'bar'
$ ^foo^bar - run last command, replacing 'foo' with 'bar'
$ cp filename{,.bak} - create a .bak copy of file
<Ctrl-x> <Ctrl-e> - open $EDITOR to edit current command line; upon exit, content will be run
$ <space>command - run the command without saving it in history
$ > file.txt - empty a file
<Alt+.> - put last argument on cursor place; repeat to cycle through
$ (cd /tmp && ls) - jump to /tmp, execute ls, and jump back here
$ pushd /tmp .... $ popd - goes to /tmp, remembering current location; at later time, popd will bring you back
$ \command - run command without using aliases
$ rm !(pattern|pattern) - delete files that do not match given patterns
$ find . -mtime +5 -exec rm {} \; - delete all files in this folder that are older than 5 days
<Ctrl-u> ... <Ctrl-y> - Ctrl+u will cut current command, then you can do whatever you need, and Ctrl+y will paste it back
$ date -d @1234567890 - convert unix timestamp to human-readable</pre>
* ''Informations''
<pre>$ mount | column -t - pretty-print mount table
$ cat /etc/issue - display distro name</pre>
* ''Small Tricks''
<pre>$ echo $(printf '%q' "A\"B") - prints escaped string (in this case, > A\"B <)
$ mount -t tmpfs tmpfs /mnt -o size=1024m - mount temporary RAM partition (fast access, non-persistent!)
$ disown -a && exit - exit the shell while keeping all running processes alive
$ ssh -t remote_host screen -r - directly connect to a remote screen process
$ for I in $(mysql -e 'show databases' -s --skip-column-names); do mysqldump $I | gzip > "$I.sql.gz"; done
- backup all databases into individual files
$ fuser -k filename - kill process that is locking given file
$ readom dev=/dev/scd0 f=/path/to/image.iso - create CD/DVD ISO image
$ diff -rq folder-1 folder-2 - compare 2 folders (files missing or differing)
$ mkdir -p a/deep/dir/tree - create nested directories fast
$ mount file.iso /mnt/iso -o loop - mount ISO file as drive /mnt/iso (dir has to preexist)
$ sudo touch /forcefsck - force a file system check on the next reboot
$ ls | while read f; do process_file "$f"; done - iterate through files even if they contain space in name
$ while read f; do process_file "$f"; done < <(ls) - iterates through files, but does not create sub-process (outside scope is visible)</pre>
* ''lsof''
<pre>$ lsof -nPi - list network connections
$ netstat -tlnp - print all listening ports with PID's (run as root to see all process names)
$ lsof -c gnome-terminal - list files opened by given command
$ lsof +D [folder] - discover all open files in the [folder]
$ lsof -p [pid] - list files opened by given PID
$ lsof -i :8080 - list programs that have port 8080 open ("who to kill" if you get "port already in use")
$ if [ `lsof -- file | wc -l` -ne 0 ] ... - tests if file is open by some process</pre>

=== [[Short Notes on C/C++|C/C++]] ===

* How to clean up after child thread
* Non-blocking IO using sockets
* Shared Libraries on Linux+GCC
* Singleton
* <tt>std::move</tt> Semantics
* Converting between <tt>std::string</tt> and <tt>std::wstring</tt> in C++11

=== cURL (in terminal) ===

Use '''-s''' option to make <tt>curl</tt> silent.

* POST to server using cURL:
<pre>curl --data "key1=value1&key2=value2" http://www.example.com/post.php</pre>
* Upload to FTP server (with credentials), having connection timeout 60 seconds, and 120 seconds maximum operation time:
<pre>curl --connect-timeout 60 -m 120 -T file.ext "ftp://12.34.56.78/folder" --user user:pass</pre>
* Download from FTP server (with credentials):
<pre>curl --user user:pass "ftp://12.34.56.78/folder/file.ext" -o ~/Downloads/file.ext</pre>

=== [[Small Docker Notes|Docker]] ===

=== [[Small elasticsearch Notes|ElasticSearch]] ===

* Installation
* What You Get
* Indexing, Routing, etc.
* Deleting
* Getting and Multi-Getting
* Updating
* Search and Query DSL

=== [[Short Notes on ffmpeg|ffmpeg]] ===

* ''process all files, may include spaces in file name:''
ls *.avi | while read f; do ffmpeg -i "$f" ''...params...'' "$f.mkv"; done
* ''if you're having problems with input format, passing video through mpeg typically helps:''
ffmpeg -i file.mkv -qscale 2 -f mpeg - | ffmpeg -i - ''...params...'' file.avi
* ''downmix 5.1 dolby to normal stereo:''
ffmpeg -i file.mkv ''...params...'' -acodec libmp3lame -ar 44100 -ac 2 file.avi
* ''add an overlay image to the video:''
ffmpeg -i input.mp4 -i image.png -filter_complex "[0:v][1:v] overlay=''x'':''y'':enable='between(t,0,20)'" -pix_fmt yuv420p -c:a copy output.mp4
This will add ''image.png'' as an overlay at position ''x'', ''y'' over the ''imput.mp4'' video, for the first 20 seconds.
* ''fade in and out, first and last 25 frames of a 1000 frame video''
ffmpeg -i input.mp4 -vf "fade=in:0:25,fade=out:975:25" -acodec copy out.mp4

==== [[Short Notes on ffmpeg|more on ffmpeg]] ====

* Rip DVD into FLV (or anything else)
* Video File with Still Image and Audio
* Concatenate Videos

=== [[Short Notes on Flask and Flask-RestPlus|Flask and Flask-RestPlus]] ===

=== [[Short Notes on git|git]] ===

=== [[Small Kyoto Cabinet and Kyoto Tycoon Notes|Kyoto Cabinet and Kyoto Tycoon]] ===

* Setup and Install Kyoto Cabinet
* Setup and Install Kyoto Tycoon
* KT's RPC API Reference

=== [[Short Notes on Linux Administration|Linux Administration]] ===

* Reset Compiz to Default Settings
* Enable and Start <tt>sendmail</tt>
* Installing <tt>sendmail</tt>
* Sending Emails using <tt>sendmail</tt>
* Find and Mark Bad Blocks
* Using TLP to optimize battery lifetime

=== [[Short Notes on MAC OSX|MAC OSX]] ===

* Asynchronous Execution
** ''NSThread''
** ''performSelectorInBackground''
** ''dispatch_async''
* ''MacPorts''
$ sudo port selfupdate
$ sudo port upgrade outdated
$ sudo port uninstall inactive
$ (just once) sudo port install port_cutleaves
$ sudo port_cutleaves

=== [[Small MongoDB Notes|MongoDB]] ===

* Installation on Ubuntu
* Troubleshooting on Desktop
* Use in PHP via PECL driver

=== [[Short Notes on MySQL|MySQL]] ===

* Error 1045: Access denied
* The <tt>insert ... on duplicate key update</tt> Pitfall

=== [[Short Notes on PDF|PDF]] ===

* Merging PDF Files using GhostScript
* Cutting PDF Files using GhostScript
* Converting to PDF using GhostScript

=== [[Short Notes on PHP|PHP]] ===

* Convert Accented Characters to Non-Accented
* Upload a File using cURL
* HTTP Post in Pure PHP (without cURL)
* Prevent Client Disconnect from Killing PHP
* Disconnect Client and Continue Processing

=== [[Short Notes on Python|Python]] ===

* Timing, and memory, on Linux
* Importing Files
* uWSGI, nginx, Flask
* Decorators
* virtualenv
* SQLAlchemy

=== [[RedBeanPHP Cheat Sheet]] ===

=== [[Short Notes on RESTful APIs|RESTful APIs]] ===

=== [[Short Notes on Security|Security]] ===

* Password Generators
* SSH Access using pubkey's Authentication
* Using RSync together with SSH

=== [[Short Notes on SVN|Subversion/SVN]] ===

* Simple Guide to SVN Client
* Creating a SVN Repository
* Moving a SVN Repository
* Change Repository URL (Subversion 1.7+)
* Setup a SVN Server on a Shared Hosting with no ''root'' Access
* creating and applying patch (the <tt>--diff-cmd</tt> is necessary in case you set some graphical diff-viewer in <tt>~/.subversion/config</tt>):
<pre>cd root_of_project
svn di --diff-cmd diff > ~/patch.diff
cd root_of_other_project
patch -p0 -i ~/patch.diff</pre>
* ''svn:externals'' - pull ''repo1'' into ''dir1'', and ''repo2'' into ''dir2'':
<pre>echo -e 'dir1 svn://repo1\ndir2 svn://repo2' > repos.txt
svn propset svn:externals . -F repos.txt</pre>
validate with " ''svn propget svn:externals .'' " .

=== Ubuntu / Debian / Gnome etc. ===

* restrict gnome app switcher to current workspace only:
<pre>gsettings set org.gnome.shell.app-switcher current-workspace-only true</pre>
* restore or undelete (config) file or directory:
<pre># 1. check what package file or folder belongs to
$ dpkg -S /path/my.conf
# 2.a replace the config file; if it exists, will offer diff
$ sudo apt-get -o Dpkg::Options::="--force-confask" install --reinstall <package-name>
# 2.b restore the directory
$ sudo apt-get -o Dpkg::Options::="--force-confask" install --reinstall $(dpkg -S /etc/some/directory | sed 's/,//g; s/:.*//')</pre>

=== [[Short Notes on ViM|ViM]] ===

* Delete all lines matching a pattern
* Directly type HTML entities
* Reformat a plain text paragraph
<pre>$ vim -x filename - open filename for editing, and save it encrypted (will prompt for a key)
:X - set encryption for current file; use ':set key=' to disable encryption
- put "set cm=blowfish" in your .vimrc to enable safer encryption by default
$ vim +10 [file] ; vim +/hello [file] - open [file] at line 10, or at first occurrence of pattern 'hello' in file
:w !sudo tee % - save file via `sudo` (as root)</pre>

=== [[Short Notes on Wavelets|Wavelets]] ===

* Integer Haar Wavelets in Python

=== [[Short Notes on Web|Web - HTML, CSS, JS, jQuery, etc.]] ===

* Multiple Submit Buttons and/or Multiple Forms on Page
* Make 'float' Elements Appear One under Another
* Upload a File via Form
* Override the IE's Problem with <tt>width</tt>
* Make an Element Fill In All the Remaining Space

=== Other Short Notes ===

==== [[Rip Video DVD that is not Finalized]] ====

==== [[Install Apache and PHP to do Secure h264 Pseudo Streaming]] ====

==== HTTP Response Codes ====

This is not a definitive list, it's just stuff I use the most, and always forget which is which.

* '''100''' - ''Continue''
* '''200''' - ''OK''
* '''201''' - ''Created''
* '''202''' - ''Accepted''
* '''204''' - ''No Content''
* '''301''' - ''Moved Permanently''
* '''302''' - ''Found'' (I typically use this instead of the '''307''' "''Temporary Redirect''")
* '''304''' - ''Not Modified''
* '''400''' - ''Bad Request''
* '''401''' - ''Unauthorized''
* '''403''' - ''Forbidden''
* '''404''' - ''Not Found''
* '''405''' - ''Method Not Allowed'' (using GET on POST calls, etc. - must include "Allow" response header)
* '''406''' - ''Not Acceptable'' (mismatch of response format and "Accept" headers)
* '''500''' - ''Internal Server Error''
* '''501''' - ''Not Implemented''
* '''503''' - ''Service Unavailable''

Main Page

2019-11-18T16:18:21Z

Admin: /* Python */

Welcome to the '''PaskvilWiki'''!

The future shape of this wiki is unknown for now, let's hope it'll be something good...

This is partially a private wiki - viewing is allowed to all, but editing is allowed only to registered users.

This is more of a prevention than denial - I don't have time to monitor and moderate edits; - you're more than welcome to request registration, all contributors are very much welcome!

Just write me at josef.at.paskvil.dot.com.

== References and Tutorials ==

=== [[emacs and slime reference|Emacs and Slime Reference]] ===

=== [[Small jQuery Reference]] ===

=== [[Small jQuery Cheat Sheet]] ===

=== [[Small SPARQL, RDQL, etc. Cheat Sheet]] ===

== Short Notes - Tips and Tricks ==

=== [[Small 4store How-To|4store]] ===

* setup and installation
* running 4store
* data import
* data import and querying from Python
* Turtle file printer (in Python)

=== [[Short Notes on Apache|Apache]] ===

* Note on serving of local files
* Self-Signed SSL Certificate
* Enable SSL/HTTPS in Apache
* <tt>.htaccess</tt> and <tt>mod_rewrite</tt> Tricks

=== [[Short Notes on AWS|AWS]] ===

* Debugging (lack of) connection to instance
* Authorization header being removed by ElasticBeanstalk

=== [[Short Notes on Bash|Bash]] ===

* Extensions and File Name
* File Test Operators
* ''Commands and Shortcuts''
<pre>$ sudo !! - run last command as root
$ !!:gs/foo/bar - run last command, replacing 'foo' with 'bar'
$ ^foo^bar - run last command, replacing 'foo' with 'bar'
$ cp filename{,.bak} - create a .bak copy of file
<Ctrl-x> <Ctrl-e> - open $EDITOR to edit current command line; upon exit, content will be run
$ <space>command - run the command without saving it in history
$ > file.txt - empty a file
<Alt+.> - put last argument on cursor place; repeat to cycle through
$ (cd /tmp && ls) - jump to /tmp, execute ls, and jump back here
$ pushd /tmp .... $ popd - goes to /tmp, remembering current location; at later time, popd will bring you back
$ \command - run command without using aliases
$ rm !(pattern|pattern) - delete files that do not match given patterns
$ find . -mtime +5 -exec rm {} \; - delete all files in this folder that are older than 5 days
<Ctrl-u> ... <Ctrl-y> - Ctrl+u will cut current command, then you can do whatever you need, and Ctrl+y will paste it back
$ date -d @1234567890 - convert unix timestamp to human-readable</pre>
* ''Informations''
<pre>$ mount | column -t - pretty-print mount table
$ cat /etc/issue - display distro name</pre>
* ''Small Tricks''
<pre>$ echo $(printf '%q' "A\"B") - prints escaped string (in this case, > A\"B <)
$ mount -t tmpfs tmpfs /mnt -o size=1024m - mount temporary RAM partition (fast access, non-persistent!)
$ disown -a && exit - exit the shell while keeping all running processes alive
$ ssh -t remote_host screen -r - directly connect to a remote screen process
$ for I in $(mysql -e 'show databases' -s --skip-column-names); do mysqldump $I | gzip > "$I.sql.gz"; done
- backup all databases into individual files
$ fuser -k filename - kill process that is locking given file
$ readom dev=/dev/scd0 f=/path/to/image.iso - create CD/DVD ISO image
$ diff -rq folder-1 folder-2 - compare 2 folders (files missing or differing)
$ mkdir -p a/deep/dir/tree - create nested directories fast
$ mount file.iso /mnt/iso -o loop - mount ISO file as drive /mnt/iso (dir has to preexist)
$ sudo touch /forcefsck - force a file system check on the next reboot
$ ls | while read f; do process_file "$f"; done - iterate through files even if they contain space in name
$ while read f; do process_file "$f"; done < <(ls) - iterates through files, but does not create sub-process (outside scope is visible)</pre>
* ''lsof''
<pre>$ lsof -nPi - list network connections
$ netstat -tlnp - print all listening ports with PID's (run as root to see all process names)
$ lsof -c gnome-terminal - list files opened by given command
$ lsof +D [folder] - discover all open files in the [folder]
$ lsof -p [pid] - list files opened by given PID
$ lsof -i :8080 - list programs that have port 8080 open ("who to kill" if you get "port already in use")
$ if [ `lsof -- file | wc -l` -ne 0 ] ... - tests if file is open by some process</pre>

=== [[Short Notes on C/C++|C/C++]] ===

* How to clean up after child thread
* Non-blocking IO using sockets
* Shared Libraries on Linux+GCC
* Singleton
* <tt>std::move</tt> Semantics
* Converting between <tt>std::string</tt> and <tt>std::wstring</tt> in C++11

=== cURL (in terminal) ===

Use '''-s''' option to make <tt>curl</tt> silent.

* POST to server using cURL:
<pre>curl --data "key1=value1&key2=value2" http://www.example.com/post.php</pre>
* Upload to FTP server (with credentials), having connection timeout 60 seconds, and 120 seconds maximum operation time:
<pre>curl --connect-timeout 60 -m 120 -T file.ext "ftp://12.34.56.78/folder" --user user:pass</pre>
* Download from FTP server (with credentials):
<pre>curl --user user:pass "ftp://12.34.56.78/folder/file.ext" -o ~/Downloads/file.ext</pre>

=== [[Small Docker Notes|Docker]] ===

=== [[Small elasticsearch Notes|ElasticSearch]] ===

* Installation
* What You Get
* Indexing, Routing, etc.
* Deleting
* Getting and Multi-Getting
* Updating
* Search and Query DSL

=== [[Short Notes on ffmpeg|ffmpeg]] ===

* ''process all files, may include spaces in file name:''
ls *.avi | while read f; do ffmpeg -i "$f" ''...params...'' "$f.mkv"; done
* ''if you're having problems with input format, passing video through mpeg typically helps:''
ffmpeg -i file.mkv -qscale 2 -f mpeg - | ffmpeg -i - ''...params...'' file.avi
* ''downmix 5.1 dolby to normal stereo:''
ffmpeg -i file.mkv ''...params...'' -acodec libmp3lame -ar 44100 -ac 2 file.avi
* ''add an overlay image to the video:''
ffmpeg -i input.mp4 -i image.png -filter_complex "[0:v][1:v] overlay=''x'':''y'':enable='between(t,0,20)'" -pix_fmt yuv420p -c:a copy output.mp4
This will add ''image.png'' as an overlay at position ''x'', ''y'' over the ''imput.mp4'' video, for the first 20 seconds.
* ''fade in and out, first and last 25 frames of a 1000 frame video''
ffmpeg -i input.mp4 -vf "fade=in:0:25,fade=out:975:25" -acodec copy out.mp4

==== [[Short Notes on ffmpeg|more on ffmpeg]] ====

* Rip DVD into FLV (or anything else)
* Video File with Still Image and Audio
* Concatenate Videos

=== [[Short Notes on Flask and Flask-RestPlus|Flask and Flask-RestPlus]] ===

=== [[Short Notes on git|git]] ===

=== [[Small Kyoto Cabinet and Kyoto Tycoon Notes|Kyoto Cabinet and Kyoto Tycoon]] ===

* Setup and Install Kyoto Cabinet
* Setup and Install Kyoto Tycoon
* KT's RPC API Reference

=== [[Short Notes on Linux Administration|Linux Administration]] ===

* Reset Compiz to Default Settings
* Enable and Start <tt>sendmail</tt>
* Installing <tt>sendmail</tt>
* Sending Emails using <tt>sendmail</tt>
* Find and Mark Bad Blocks

=== [[Short Notes on MAC OSX|MAC OSX]] ===

* Asynchronous Execution
** ''NSThread''
** ''performSelectorInBackground''
** ''dispatch_async''
* ''MacPorts''
$ sudo port selfupdate
$ sudo port upgrade outdated
$ sudo port uninstall inactive
$ (just once) sudo port install port_cutleaves
$ sudo port_cutleaves

=== [[Small MongoDB Notes|MongoDB]] ===

* Installation on Ubuntu
* Troubleshooting on Desktop
* Use in PHP via PECL driver

=== [[Short Notes on MySQL|MySQL]] ===

* Error 1045: Access denied
* The <tt>insert ... on duplicate key update</tt> Pitfall

=== [[Short Notes on PDF|PDF]] ===

* Merging PDF Files using GhostScript
* Cutting PDF Files using GhostScript
* Converting to PDF using GhostScript

=== [[Short Notes on PHP|PHP]] ===

* Convert Accented Characters to Non-Accented
* Upload a File using cURL
* HTTP Post in Pure PHP (without cURL)
* Prevent Client Disconnect from Killing PHP
* Disconnect Client and Continue Processing

=== [[Short Notes on Python|Python]] ===

* Timing, and memory, on Linux
* Importing Files
* uWSGI, nginx, Flask
* Decorators
* virtualenv
* SQLAlchemy

=== [[RedBeanPHP Cheat Sheet]] ===

=== [[Short Notes on RESTful APIs|RESTful APIs]] ===

=== [[Short Notes on Security|Security]] ===

* Password Generators
* SSH Access using pubkey's Authentication
* Using RSync together with SSH

=== [[Short Notes on SVN|Subversion/SVN]] ===

* Simple Guide to SVN Client
* Creating a SVN Repository
* Moving a SVN Repository
* Change Repository URL (Subversion 1.7+)
* Setup a SVN Server on a Shared Hosting with no ''root'' Access
* creating and applying patch (the <tt>--diff-cmd</tt> is necessary in case you set some graphical diff-viewer in <tt>~/.subversion/config</tt>):
<pre>cd root_of_project
svn di --diff-cmd diff > ~/patch.diff
cd root_of_other_project
patch -p0 -i ~/patch.diff</pre>
* ''svn:externals'' - pull ''repo1'' into ''dir1'', and ''repo2'' into ''dir2'':
<pre>echo -e 'dir1 svn://repo1\ndir2 svn://repo2' > repos.txt
svn propset svn:externals . -F repos.txt</pre>
validate with " ''svn propget svn:externals .'' " .

=== Ubuntu / Debian / Gnome etc. ===

* restrict gnome app switcher to current workspace only:
<pre>gsettings set org.gnome.shell.app-switcher current-workspace-only true</pre>
* restore or undelete (config) file or directory:
<pre># 1. check what package file or folder belongs to
$ dpkg -S /path/my.conf
# 2.a replace the config file; if it exists, will offer diff
$ sudo apt-get -o Dpkg::Options::="--force-confask" install --reinstall <package-name>
# 2.b restore the directory
$ sudo apt-get -o Dpkg::Options::="--force-confask" install --reinstall $(dpkg -S /etc/some/directory | sed 's/,//g; s/:.*//')</pre>

=== [[Short Notes on ViM|ViM]] ===

* Delete all lines matching a pattern
* Directly type HTML entities
* Reformat a plain text paragraph
<pre>$ vim -x filename - open filename for editing, and save it encrypted (will prompt for a key)
:X - set encryption for current file; use ':set key=' to disable encryption
- put "set cm=blowfish" in your .vimrc to enable safer encryption by default
$ vim +10 [file] ; vim +/hello [file] - open [file] at line 10, or at first occurrence of pattern 'hello' in file
:w !sudo tee % - save file via `sudo` (as root)</pre>

=== [[Short Notes on Wavelets|Wavelets]] ===

* Integer Haar Wavelets in Python

=== [[Short Notes on Web|Web - HTML, CSS, JS, jQuery, etc.]] ===

* Multiple Submit Buttons and/or Multiple Forms on Page
* Make 'float' Elements Appear One under Another
* Upload a File via Form
* Override the IE's Problem with <tt>width</tt>
* Make an Element Fill In All the Remaining Space

=== Other Short Notes ===

==== [[Rip Video DVD that is not Finalized]] ====

==== [[Install Apache and PHP to do Secure h264 Pseudo Streaming]] ====

==== HTTP Response Codes ====

This is not a definitive list, it's just stuff I use the most, and always forget which is which.

* '''100''' - ''Continue''
* '''200''' - ''OK''
* '''201''' - ''Created''
* '''202''' - ''Accepted''
* '''204''' - ''No Content''
* '''301''' - ''Moved Permanently''
* '''302''' - ''Found'' (I typically use this instead of the '''307''' "''Temporary Redirect''")
* '''304''' - ''Not Modified''
* '''400''' - ''Bad Request''
* '''401''' - ''Unauthorized''
* '''403''' - ''Forbidden''
* '''404''' - ''Not Found''
* '''405''' - ''Method Not Allowed'' (using GET on POST calls, etc. - must include "Allow" response header)
* '''406''' - ''Not Acceptable'' (mismatch of response format and "Accept" headers)
* '''500''' - ''Internal Server Error''
* '''501''' - ''Not Implemented''
* '''503''' - ''Service Unavailable''

Short Notes on Python

2019-11-18T16:18:04Z

Admin: /* virtualenv */