Archive

Author Archive

The way to fully automated releases in open source projects

June 24, 2020 Leave a comment
Categories: Interesting

Best practices for managing and storing secrets including API keys and other credentials

June 18, 2020 Leave a comment

Storing and managing secrets like API keys and other credentials can be challenging, even the most careful policies can sometimes be circumvented in exchange for convenience. We have compiled a list of some of the best practices to help keep secrets and credentials safe. Secrets management doesn’t have a one-size-fits-all approach so this list considers multiple perspectives so you can be informed in deciding to, or not to, implement strategies.

https://blog.gitguardian.com/secrets-api-management/

Source:
programmerweekly.com

Categories: Interesting

heroku – alembic

May 2, 2020 Leave a comment

Problem:
Alembic migrations are not run automatically on heroku. On first install of the application, attempting to run $heroku run alembic upgrade head --app application-name results in an error as it appears that tables are build from the models that were defined in the application.

$ heroku run alembic history --app application-name
$ heroku run alembic downgrade -1 --app application-name fails with

INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO [alembic.runtime.migration] Will assume transactional DDL.
ERROR [alembic.util.messaging] Relative revision -1 didn't produce 1 migrations
FAILED: Relative revision -1 didn't produce 1 migrations

$ heroku run alembic downgrade 017e517ce319 --app application-name fails with
INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO [alembic.runtime.migration] Will assume transactional DDL.
ERROR [alembic.util.messaging] Destination 017e517ce319 is not a valid downgrade target from current head(s)
FAILED: Destination 017e517ce319 is not a valid downgrade target from current head(s)

On checking the alembic version in the database
$ heroku pg:psql DATABASE_URL --app application-name
application-name::DATABASE=>select * from alembic_version;
version_num
-------------
(0 rows)

application-name::DATABASE=> update alembic_version set version_num = '017e517ce319';
UPDATE 0
application-name::DATABASE=> select * from alembic_version;
version_num
-------------
(0 rows)

Solution:
To start using alembic I needed to drop all tables and run alembic. Doing a dump of the db, running alembic, then reading back the dump should also work.

1. Clear all data
$ heroku run pg:reset DATABASE_URL --app application-name
2. Run alembic
3. Moving forward, always manually run alembic. (You could add this to your deploy step as well, depends on your needs.)
4. alembic downgrade,upgrade should now work.

Categories: postgresql Tags: , , ,

https://use-the-index-luke.com/

May 1, 2020 Leave a comment

A site explaining SQL indexing to developers—no crap about administration.

SQL indexing is the most effective tuning method—yet it is often neglected during development. Use The Index, Luke explains SQL indexing from grounds up and doesn’t stop at ORM tools like Hibernate.

Use The Index, Luke is the free web-edition of SQL Performance Explained. If you like this site, consider getting the book. Also have a look at the shop for other cool stuff that supports this site.

https://use-the-index-luke.com/

Categories: Interesting

ImportError: attempted relative import with no known parent package

April 30, 2020 Leave a comment

You are trying to use relative imports in python, but you encounter one of the exceptions ImportError: attempted relative import with no known parent package. Yes, relative imports can be confusing from time to time. In this article, we see the method used by python interpreter to resolve the relative imports and how we can fix this issue.

https://napuzba.com/a/import-error-relative-no-parent

Categories: Interesting

Install black on VIM

April 26, 2020 Leave a comment

What is black?

Black is the uncompromising Python code formatter. By using it, you agree to cede control over minutiae of hand-formatting. In return, Black gives you speed, determinism, and freedom from pycodestyle nagging about formatting. You will save time and mental energy for more important matters.

Blackened code looks the same regardless of the project you’re reading. Formatting becomes transparent after a while and you can focus on the content instead.

Black makes code review faster by producing the smallest diffs possible.

From https://github.com/psf/black

Installation steps
1. Deactivate any active virtual environments you may have running.

$ deactivate

Failure to do may lead to the following errors after completing all the installation steps on the github page.

When running :Black

Traceback (most recent call last):
  File "", line 1, in 
NameError: name 'Black' is not defined

opening a *.py file

Error detected while processing ~/.vim/plugged/black/plugin/black.vim:
line  124:
Traceback (most recent call last):
  File "", line 56, in 
ModuleNotFoundError: No module named 'black'

2. Install a plugin manager. Eg Vundle

$ git clone https://github.com/VundleVim/Vundle.vim.git ~/.vim/bundle/Vundle.vim

3. Edit .vimrc to install plugin.
With Vundle

$ less .vimrc
set nocompatible              " be iMproved, required
filetype off                  " required

" set the runtime path to include Vundle and initialize
set rtp+=~/.vim/bundle/Vundle.vim
call vundle#begin()

" let Vundle manage Vundle, required
Plugin 'VundleVim/Vundle.vim'
Plugin 'psf/black'

" All of your Plugins must be added before the following line
call vundle#end()            " required
filetype plugin indent on    " required

" Specify the virtual environment path for black
let g:black_virtualenv='~/.vim/black'
" Run black on save
autocmd BufWritePre *.py execute ':Black'

4. Install the black plugin

$ vim test.py
:PluginInstall

You may get the following error

Done!
Please wait, one time setup for Black.
Creating a virtualenv in /home/vagrant/.vim/black...
(this path can be customized in .vimrc by setting g:black_virtualenv)
The virtual environment was not created successfully because ensurepip is not
available.  On Debian/Ubuntu systems, you need to install the python3-venv
package using the following command.
    apt-get install python3-venv
You may need to use sudo with that command.  After installing the python3-venv
package, recreate your virtual environment.
Failing command: ['/home/vagrant/.vim/black/bin/python3', '-Im', 'ensurepip', '--upgrade', '--default-pip']
Error detected while processing /home/vagrant/.vim/bundle/black/plugin/black.vim:
line  194:
E880: Can't handle SystemExit of python exception in vim
Press ENTER or type command to continue

5. Install python3-venv

$ sudo apt-get install python3-venv

6. Recreate the virtual environment

$ rm -rf .vim/black
$ vim test.py
vim test.py
Please wait, one time setup for Black.
Creating a virtualenv in /home/vagrant/.vim/black...
(this path can be customized in .vimrc by setting g:black_virtualenv)
Installing Black with pip...
DONE! You are all set, thanks for waiting ✨ 🍰 ✨
Pro-tip: to upgrade Black in the future, use the :BlackUpgrade command and restart Vim.
Press ENTER or type command to continue

7. Last step of installation

vagrant@vagrant:~$ cd .vim/bundle/black/
vagrant@vagrant:~/.vim/bundle/black$ git checkout origin/stable -b stable
Branch 'stable' set up to track remote branch 'stable' from 'origin'.
Switched to a new branch 'stable'
vagrant@vagrant:~/.vim/bundle/black$ cd ~

8. Test black
Add the following to test.py and write (:w)

j = [1,
2,
3
]

Black should reformat the file in place and give you

j = [1, 2, 3]

You may get the following error

Error detected while processing BufWrite Auto commands for "*.py":
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 92, in Black
  File "", line 128, in get_configs
"test.py" [New] 5L, 26C written
Press ENTER or type command to continue

Try running :Black inside test.py to see the actual error

Traceback (most recent call last):
  File "", line 1, in 
  File "", line 92, in Black
  File "", line 128, in get_configs
AttributeError: module 'black' has no attribute 'find_pyproject_toml'

The above error may occur if you skipped step 7. You could also run https://github.com/psf/black/pull/1273#issuecomment-594114067

System details
– Ubuntu 18.04.3 LTS
– VIM 8.0 with python3 support
vim test.py
:echo has(‘python3’)
1

Source:
https://github.com/psf/black#installation-and-usage
https://github.com/psf/black/issues/672
https://github.com/psf/black/pull/1273

Categories: vim Tags: , ,

Commit only part of a file in Git

April 21, 2020 Leave a comment

You can use git add –patch (or -p for short), and git will begin to break down your file into what it thinks are sensible “hunks” (portions of the file). It will then prompt you with this question:

Stage this hunk [y,n,q,a,d,/,j,J,g,s,e,?]?

Here is a description of each option:

y – stage this hunk for the next commit
n – do not stage this hunk for the next commit
q – quit; do not stage this hunk or any of the remaining hunks
a – stage this hunk and all later hunks in the file
d – do not stage this hunk or any of the later hunks in the file
g – select a hunk to go to
/ – search for a hunk matching the given regex
j – leave this hunk undecided, see next undecided hunk
J – leave this hunk undecided, see next hunk
k – leave this hunk undecided, see previous undecided hunk
K – leave this hunk undecided, see previous hunk
s – split the current hunk into smaller hunks
e – manually edit the current hunk
? – print hunk help

If the file is not in the repository yet, you can first do git add -N . Afterwards you can go on with git add -p .

Afterwards, you can use:

git diff –staged to check that you staged the correct changes
git reset -p to unstage mistakenly added hunks
git commit -v to view your commit while you edit the commit message.

Note this is far different than the git format-patch command, whose purpose is to parse commit data into a .patch files.

Reference for future: Git Tools – Interactive Staging

Source
https://stackoverflow.com/questions/1085162/commit-only-part-of-a-file-in-git

Categories: git

Deploy Any Python Project to Kubernetes

April 16, 2020 Leave a comment

As your project grows, it might get to the point that it becomes too hard to handle with just single VM or some simple SaaS solution. You can solve that by switching to more robust solution like Kubernetes. That might however, be little too complex if you are not familiar with it’s concepts or if just never used it before. So, to help you out – in this article – we will go over all you need to get yourself started and have your Python project deployed on cluster – including cluster setup, all the Kubernetes manifests and some extra automation to make your life easier down the road!

This is a follow up to previous article(s) about Automating Every Aspect of Your Python Project, so you might want check that out before reading this one.

TL;DR: Here is my repository with full source code and docs: https://github.com/MartinHeinz/python-project-blueprint

https://martinheinz.dev/blog/20

Source:
Python Weekly

Categories: Interesting, kubernetes, python Tags:

WhyTheName

April 8, 2020 Leave a comment
Categories: Interesting

Personal Security Tips — The Ubuntu Incident

March 19, 2020 Leave a comment

See https://github.com/Lissy93/personal-security-checklist for a curated checklist of 100+ tips for protecting digital security and privacy.

via Personal Security Tips — The Ubuntu Incident

Categories: Interesting