Archive

Archive for the ‘bash’ Category

Change the last line in a file

February 12, 2019 Leave a comment

Problem:
I needed to remove a comma from the last line of a file.

Solution:
Learnt that I can specify the last line with sed.

# File with commas
$ less minerals.csv
"copper",
"bronze",
"gold",
"platinum",
$ sed '$ s/,//' minerals.csv
$ less minerals.csv
"copper",
"bronze",
"gold",
"platinum"

Addresses

Sed commands can be given with no addresses, in which case the command will be executed for all input lines; with one address, in which case the command will only be executed for input lines which match that address; or with two addresses, in which case the command will be executed for all input lines which match the inclusive range of lines starting from the first address and continuing to the second address. Three things to note about address ranges: the syntax is addr1,addr2 (i.e., the addresses are separated by a comma); the line which addr1 matched will always be accepted, even if addr2 selects an earlier line; and if addr2 is a regexp, it will not be tested against the line that addr1 matched.

After the address (or address-range), and before the command, a ! may be inserted, which specifies that the command shall only be executed if the address (or address-range) does not match.

The following address types are supported:

number
Match only the specified line number.

first~step
Match every step’th line starting with line first. For example, ”sed -n 1~2p” will print all the odd-numbered lines in the input stream, and the address 2~5 will match every fifth line, starting with the second. first can be zero; in this case, sed operates as if it were equal to step. (This is an extension.)
$
Match the last line.

/regexp/
Match lines matching the regular expression regexp.
\cregexpc
Match lines matching the regular expression regexp. The c may be any character.
GNU sed also supports some special 2-address forms:
0,addr2
Start out in “matched first address” state, until addr2 is found. This is similar to 1,addr2, except that if addr2 matches the very first line of input the 0,addr2 form will be at the end of its range, whereas the 1,addr2 form will still be at the beginning of its range. This works only when addr2 is a regular expression.
addr1,+N
Will match addr1 and the N lines following addr1.
addr1,~N
Will match addr1 and the lines following addr1 until the next line whose input line number is a multiple of N.

Source:
https://stackoverflow.com/questions/3576139/sed-remove-string-only-in-the-last-line-of-the-file
https://linux.die.net/man/1/sed

Advertisements
Categories: bash, sed

Reading environment variables from a file into a Makefile

January 30, 2019 Leave a comment

Problem:
You have a bunch of environment variables that are explicitly set at the beginning of a Makefile. These variables are then used in various recipes in the Makefile. The issue was that when a developer runs the Makefile, these environment variables overwrite their local settings.

Solution:
One option is to move the environment variables out of the Makefile and use include to add them back. So the developer can use whatever values they have in their current setup as long as the names are the same.

# Makefile

foo=FOO1
bar=BAR1
baz=BAZ1

export foo
export bar
export baz

.PHONY:  target1 target2 target3

target1:
    @echo $(foo)

target2:
    @echo $(bar)

target3:
    @echo $(baz)

After splitting it up.

# .env file
foo=FOO1
bar=BAR1
baz=BAZ1

export foo
export bar
export baz
# Makefile
include .env

.PHONY:  target1 target2 target3

target1:
    @echo $(foo)

target2:
    @echo $(bar)

target3:
    @echo $(baz)

The problem with this approach is that the .env file needs to exist everywhere it will run. If you have some CI/CD setup that may not be the case. Instead have the Makefile as below and have the ENV vars set up the usual way you would in your CI/CD setup. On the local machine one can simply run $ source .env before running $ make to get things to work.

# Makefile
.PHONY:  target1 target2 target3

target1:
    @echo $(foo)

target2:
    @echo $(bar)

target3:
    @echo $(baz)

Source:
https://stackoverflow.com/questions/39005789/reading-makefile-variables-from-bash-script

Categories: bash Tags:

GNU Makefile setting env variables

January 12, 2019 Leave a comment

Problem:
Exporting env variables in Makefiles. The issue was that the environment variables were not visible across different targets. I needed the same env vars to be seen in both target1 and target2 below.

Solution:
Export the variables outside the target, eg at the top of the file.

VALUE1="myvalue1"
VALUE2="myvalue2"

export VALUE1
export VALUE2

target1:
    echo $VALUE1

target2:
    echo $VALUE2

Source:
https://stackoverflow.com/questions/2826029/passing-additional-variables-from-command-line-to-make
https://www.linuxquestions.org/questions/programming-9/exporting-makefile-variables-to-%24-shell-environment-807422/

Categories: bash, Interesting Tags: ,

Useful Linux Command Line Bash Shortcuts You Should Know

November 21, 2018 Leave a comment

In this article, we will share a number of Bash command-line shortcuts useful for any Linux user. These shortcuts allow you to easily and in a fast manner, perform certain activities such as accessing and running previously executed commands, opening an editor, editing/deleting/changing text on the command line, moving the cursor, controlling processes etc. on the command line.

Although this article will mostly benefit Linux beginners getting their way around with command line basics, those with intermediate skills and advanced users might also find it practically helpful. We will group the bash keyboard shortcuts according to categories as follows.

https://www.tecmint.com/linux-command-line-bash-shortcut-keys/

Categories: bash, Interesting

docker: Error response from daemon…

August 22, 2018 Leave a comment

Problem:
I was getting this error.

docker: Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused “exec: \”telnet\”: executable file not found in $PATH”: unknown.

$ docker run --name greeter -d --expose 5000 alpine /bin/sh -c "echo Welcome stranger | nc -lp 5000"
51b6a0b73dc1cbc9b39757dff20ada1f5638c1ec781625d9e2fc6c9cced01639
$ docker exec greeter ifconfig
eth0      Link encap:Ethernet  HWaddr 02:42:AC:11:00:02
          inet addr:172.17.0.2  Bcast:172.17.255.255  Mask:255.255.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:9 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:758 (758.0 B)  TX bytes:0 (0.0 B)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
$ docker run alpine telnet 172.17.0.2 5000
docker: Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "exec: \"telnet\": executable file not found in $PATH": unknown.
ERRO[0000] error waiting for container: context canceled
$ docker run -it alpine /bin/sh
/ # telnet 172.17.0.2 5000
/bin/sh: telnet: not found

Solution:
This error means that whatever I am trying to run does not exist. So my options are to install telnet in the container or use a different command like `nc`.

Categories: bash Tags:

port forwarding

August 17, 2018 Leave a comment

Problem:

You want to access a site with a particular IP. Eg google maps only allows a set of IPs to access. So running code from a vagrant box then accessing from the host is not so straight forward.

Solution:
Went with using ssh

$ ssh -nNT -L 3000:localhost:3000 vagrant@127.0.0.1 -p 2222

So on the host I can access a service running on a vagrant box (192.168.33.10:3000) via localhost:3000

Categories: bash

find command examples

October 24, 2017 Leave a comment

Problem:
I needed to clean up after running tests

Solution:
Using find

$ tree
.
├── bin
│   └── hello
├── hello
│   ├── hello.py
│   └── __init__.py
├── requirements.txt
├── setup.py
└── tests
    ├── __init__.py
    └── test_hello.py

3 directories, 7 files
$ python3 setup.py test                                            
running test
running egg_info
creating hello.egg-info
writing dependency_links to hello.egg-info/dependency_links.txt
writing top-level names to hello.egg-info/top_level.txt
writing hello.egg-info/PKG-INFO
writing manifest file 'hello.egg-info/SOURCES.txt'
reading manifest file 'hello.egg-info/SOURCES.txt'
writing manifest file 'hello.egg-info/SOURCES.txt'
running build_ext
test_print_hello (tests.test_hello.TestHello) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.014s

OK

Now more files are created.

$ tree
.
├── bin
│   └── hello
├── hello
│   ├── hello.py
│   ├── __init__.py
│   └── __pycache__
│       ├── hello.cpython-35.pyc
│       └── __init__.cpython-35.pyc
├── hello.egg-info
│   ├── dependency_links.txt
│   ├── PKG-INFO
│   ├── SOURCES.txt
│   └── top_level.txt
├── requirements.txt
├── setup.py
└── tests
    ├── __init__.py
    ├── __pycache__
    │   ├── __init__.cpython-35.pyc
    │   └── test_hello.cpython-35.pyc
    └── test_hello.py

6 directories, 15 files

To delete the *.pyc files and the __pycache__ and hello.egg-info folders, run the following commands.

$ find . -type d -name "hello.egg-info" | xargs rm -rf
$ find . -type d -name "__pycache__" | xargs rm -rf
$ find . -name "*.pyc" | xargs rm -f

Note:

$ find --version                                                   
find (GNU findutils) 4.7.0-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Eric B. Decker, James Youngman, and Kevin Dalley.
Features enabled: D_TYPE O_NOFOLLOW(enabled) LEAF_OPTIMISATION FTS(FTS_CWDFD) CBO(level=2) 

Source:
https://superuser.com/questions/736272/using-unixs-find-command-to-find-directories-matching-name-but-not-subdirectori#736275

Categories: bash Tags: ,