Developer Tools - Complete Solutions#
This notebook contains solutions to all exercises from the Developer Tools track.
How to Use This Notebook#
Try first: Attempt exercises before checking solutions
Understand commands: Don’t just copy - understand what each command does
Practice variations: Try different approaches and options
Build muscle memory: Type commands yourself instead of copy-pasting
Experiment safely: Use test directories and files for practice
Contents#
01 - Shell Basics#
02 - Command Line Tools#
Exercise 1: Log Analysis#
Scenario: Analyze a web server log file.
Tasks:
Count total requests
Find top 10 IP addresses
Count requests by HTTP method
Find all 404 errors
Solution#
First, let’s create a sample log file for testing:
# Create sample web server log
cat > access.log << 'EOF'
192.168.1.100 - - [01/Nov/2025:10:00:00 +0000] "GET /index.html HTTP/1.1" 200 1234
192.168.1.101 - - [01/Nov/2025:10:00:01 +0000] "POST /api/users HTTP/1.1" 201 567
192.168.1.100 - - [01/Nov/2025:10:00:02 +0000] "GET /about.html HTTP/1.1" 200 890
192.168.1.102 - - [01/Nov/2025:10:00:03 +0000] "GET /missing.html HTTP/1.1" 404 0
192.168.1.100 - - [01/Nov/2025:10:00:04 +0000] "GET /contact.html HTTP/1.1" 200 456
192.168.1.103 - - [01/Nov/2025:10:00:05 +0000] "DELETE /api/users/5 HTTP/1.1" 204 0
192.168.1.101 - - [01/Nov/2025:10:00:06 +0000] "GET /notfound.html HTTP/1.1" 404 0
192.168.1.100 - - [01/Nov/2025:10:00:07 +0000] "PUT /api/users/1 HTTP/1.1" 200 123
EOF
Now analyze it:
# Count total requests
wc -l access.log
# OR
cat access.log | wc -l
# Output: 8 access.log
# Find top 10 IP addresses
awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -10
# Output:
# 4 192.168.1.100
# 2 192.168.1.101
# 1 192.168.1.103
# 1 192.168.1.102
# Count requests by HTTP method
awk '{print $6}' access.log | sed 's/"//g' | sort | uniq -c
# Output:
# 1 DELETE
# 5 GET
# 1 POST
# 1 PUT
# Find all 404 errors
grep '" 404 ' access.log
# OR with line numbers
grep -n '" 404 ' access.log
# OR count them
grep -c '" 404 ' access.log
# Output: 2
# Extract just the URLs that returned 404
grep '" 404 ' access.log | awk '{print $7}'
# Output:
# /missing.html
# /notfound.html
Command Breakdown#
IP Address Analysis:
awk '{print $1}'- Extract first field (IP address)sort- Sort IPs alphabeticallyuniq -c- Count consecutive duplicatessort -rn- Sort numerically in reverse (highest first)head -10- Take top 10
HTTP Method Count:
awk '{print $6}'- Extract 6th field (method with quote)sed 's/"//g'- Remove quotessort | uniq -c- Count each unique method
Alternative with Modern Tools#
# If ripgrep (rg) is available
rg '" 404 ' access.log
# If bat is available (cat with syntax highlighting)
bat access.log
Exercise 2: Text Processing Pipeline#
Task: Process a text file to find the most common words.
Solution#
# Create sample text file
cat > text.txt << 'EOF'
The quick brown fox jumps over the lazy dog.
The dog was not amused by the fox.
The fox was quick and clever.
EOF
# Complete word frequency pipeline
cat text.txt | \
tr '[:upper:]' '[:lower:]' | \ # Convert to lowercase
tr -s '[:punct:][:space:]' '\n' | \ # Replace punctuation/spaces with newlines
grep -v '^$' | \ # Remove empty lines
sort | \ # Sort words
uniq -c | \ # Count unique words
sort -rn | \ # Sort by count (descending)
head -10 # Top 10 words
# Output:
# 4 the
# 3 fox
# 2 was
# 2 dog
# 1 quick
# 1 over
# 1 not
# 1 lazy
# 1 jumps
# 1 clever
# Alternative: Using awk for more control
awk '{
for(i=1; i<=NF; i++) {
word = tolower($i)
gsub(/[^a-z]/, "", word)
if(word != "") count[word]++
}
}
END {
for(word in count)
print count[word], word
}' text.txt | sort -rn | head -10
Exercise 3: Find and Process Files#
Tasks:
Find all Python files
Count lines of code in each
Find files modified in last 7 days
Find large files (>100MB)
Solution#
# Find all Python files (recursively)
find . -name "*.py" -type f
# Count lines in each Python file
find . -name "*.py" -type f -exec wc -l {} \;
# OR with filenames
find . -name "*.py" -type f -exec wc -l {} + | sort -n
# Total lines of Python code
find . -name "*.py" -type f -exec cat {} \; | wc -l
# Find files modified in last 7 days
find . -type f -mtime -7
# With details
find . -type f -mtime -7 -ls
# Find large files (>100MB)
find . -type f -size +100M
# With human-readable sizes
find . -type f -size +100M -exec ls -lh {} \;
# Find and delete old temporary files (be careful!)
find /tmp -name "*.tmp" -mtime +30 -delete
# SAFER: List first, then delete
find /tmp -name "*.tmp" -mtime +30
# If looks good:
find /tmp -name "*.tmp" -mtime +30 -delete
# Find empty directories
find . -type d -empty
Modern Alternative: fd#
# If fd is installed (faster, user-friendly alternative to find)
# Find Python files
fd '\.py$'
# Find files modified in last 7 days
fd --changed-within 7d
# Find large files
fd --size +100m
# Execute command on results
fd '\.py$' --exec wc -l
Exercise 4: Process Management#
Tasks:
List running processes
Find processes by name
Monitor system resources
Kill a process
Solution#
# List all running processes
ps aux
# OR just your processes
ps ux
# Find processes by name
ps aux | grep python
# Better: use pgrep
pgrep python
# With full command
pgrep -a python
# Monitor processes in real-time
top
# OR better: htop (if installed)
htop
# Sort by memory usage
ps aux --sort=-%mem | head -10
# Sort by CPU usage
ps aux --sort=-%cpu | head -10
# Kill a process
# 1. Find the PID
pgrep python
# 2. Kill it
kill 12345 # Replace with actual PID
# Force kill if needed
kill -9 12345
# OR
kill -SIGKILL 12345
# Kill by name
pkill python
# Force kill by name
pkill -9 python
# Kill all matching processes
killall python
# Check if process is still running
ps aux | grep 12345
Signal Types#
# Gentle signals (allow cleanup)
kill -SIGTERM 12345 # Default, polite termination
kill -SIGINT 12345 # Interrupt (Ctrl+C)
# Force kill (no cleanup)
kill -SIGKILL 12345 # Cannot be caught or ignored
# Other useful signals
kill -SIGHUP 12345 # Reload configuration
kill -SIGSTOP 12345 # Pause process
kill -SIGCONT 12345 # Resume process
Exercise 5: Advanced Text Processing with sed and awk#
Tasks:
Replace text in files
Extract specific columns
Process CSV files
Generate reports
Solution#
# Create sample data
cat > employees.csv << 'EOF'
name,department,salary,years
Alice,Engineering,120000,5
Bob,Marketing,80000,3
Charlie,Engineering,95000,2
Diana,Sales,110000,7
Eve,Engineering,105000,4
EOF
# Replace text in file (sed)
# Replace 'Engineering' with 'Tech'
sed 's/Engineering/Tech/g' employees.csv
# In-place replacement (with backup)
sed -i.bak 's/Engineering/Tech/g' employees.csv
# Extract specific columns (awk)
# Print names and salaries
awk -F',' 'NR > 1 {print $1, $3}' employees.csv
# Output:
# Alice 120000
# Bob 80000
# Charlie 95000
# Diana 110000
# Eve 105000
# Calculate total and average salary
awk -F',' '
NR > 1 {
total += $3
count++
}
END {
print "Total:", total
print "Average:", total/count
}' employees.csv
# Output:
# Total: 510000
# Average: 102000
# Filter by department
awk -F',' '$2 == "Engineering"' employees.csv
# Find highest paid employee
awk -F',' '
NR > 1 {
if ($3 > max) {
max = $3
name = $1
}
}
END {
print "Highest paid:", name, "-", max
}' employees.csv
# Output: Highest paid: Alice - 120000
# Generate formatted report
awk -F',' '
BEGIN {
print "Employee Report"
print "==============="
printf "%-15s %-15s %10s %7s\n", "Name", "Department", "Salary", "Years"
print "------------------------------------------------------"
}
NR > 1 {
printf "%-15s %-15s %10s %7s\n", $1, $2, "$"$3, $4
}
END {
print "======================================================"
}' employees.csv
Common sed Patterns#
# Delete lines
sed '3d' file.txt # Delete line 3
sed '1,5d' file.txt # Delete lines 1-5
sed '/pattern/d' file.txt # Delete lines matching pattern
# Insert/append
sed '2i\New line' file.txt # Insert before line 2
sed '2a\New line' file.txt # Append after line 2
# Multiple operations
sed -e 's/old/new/g' -e 's/foo/bar/g' file.txt
# Regular expressions
sed 's/[0-9]\{3\}-[0-9]\{4\}/XXX-XXXX/g' file.txt # Mask phone numbers
03 - Git Essentials#
Exercise 1: First Repository#
Tasks:
Create a new project folder
Initialize Git repository
Create README.md
Add and commit the file
Solution#
# 1. Create project folder
mkdir my-first-project
cd my-first-project
# 2. Initialize Git repository
git init
# Output: Initialized empty Git repository in /path/to/my-first-project/.git/
# Verify
ls -la
# You should see .git directory
# Check status
git status
# Output: On branch main (or master)
# No commits yet
# 3. Create README.md
cat > README.md << 'EOF'
# My First Project
This is my first Git repository!
## About
Learning Git fundamentals:
- Initializing repositories
- Adding and committing files
- Working with branches
## Usage
More content coming soon!
EOF
# Check status again
git status
# Output: Untracked files:
# README.md
# 4. Add file to staging area
git add README.md
# OR add all files
git add .
# Check status
git status
# Output: Changes to be committed:
# new file: README.md
# Commit with message
git commit -m "Initial commit: Add README.md"
# Output: [main (root-commit) abc1234] Initial commit: Add README.md
# 1 file changed, 15 insertions(+)
# View commit history
git log
# OR one line format
git log --oneline
# View what was changed
git show
Best Practices#
Commit Messages:
# Good commit messages
git commit -m "Add user authentication feature"
git commit -m "Fix bug in payment processing"
git commit -m "Update documentation for API endpoints"
# Bad commit messages
git commit -m "stuff" # ❌ Not descriptive
git commit -m "changes" # ❌ Too vague
git commit -m "asdfasdf" # ❌ Meaningless
Configure Git (one-time setup):
git config --global user.name "Your Name"
git config --global user.email "your.email@example.com"
git config --global init.defaultBranch main
Exercise 2: Branching and Merging#
Tasks:
Create a feature branch
Make changes on the branch
Merge back to main
Delete the feature branch
Solution#
# Check current branch
git branch
# Output: * main
# 1. Create and switch to feature branch
git checkout -b feature/add-about-page
# OR using newer syntax
git switch -c feature/add-about-page
# Verify you're on the new branch
git branch
# Output:
# main
# * feature/add-about-page
# 2. Make changes on the branch
cat > about.md << 'EOF'
# About This Project
This project demonstrates Git workflow.
## Features
- Version control
- Branching and merging
- Collaboration
EOF
# Add and commit
git add about.md
git commit -m "Add about page"
# Make another change
echo "\n## Contact\nEmail: example@example.com" >> about.md
git add about.md
git commit -m "Add contact info to about page"
# View branch history
git log --oneline
# 3. Merge back to main
# First, switch to main
git checkout main
# OR
git switch main
# Merge feature branch
git merge feature/add-about-page
# Output: Updating abc1234..def5678
# Fast-forward
# about.md | 10 ++++++++++
# View merged history
git log --oneline --graph --all
# 4. Delete the feature branch
git branch -d feature/add-about-page
# Output: Deleted branch feature/add-about-page (was def5678)
# Verify branch is gone
git branch
# Output: * main
Handling Merge Conflicts#
# If merge conflict occurs:
git merge feature-branch
# Output: Auto-merging file.txt
# CONFLICT (content): Merge conflict in file.txt
# Check status
git status
# Shows conflicted files
# Open conflicted file and look for:
# <<<<<<< HEAD
# Your changes
# =======
# Their changes
# >>>>>>> feature-branch
# Edit file to resolve conflict, then:
git add file.txt
git commit -m "Merge feature-branch and resolve conflicts"
# OR abort merge
git merge --abort
Exercise 3: Working with Remote Repositories#
Tasks:
Add remote repository
Push changes to remote
Pull updates from remote
Handle divergent branches
Solution#
# 1. Add remote repository
# First, create repo on GitHub/GitLab, then:
git remote add origin https://github.com/username/my-first-project.git
# Verify remote
git remote -v
# Output:
# origin https://github.com/username/my-first-project.git (fetch)
# origin https://github.com/username/my-first-project.git (push)
# 2. Push changes to remote
# First push (set upstream)
git push -u origin main
# Output: Enumerating objects: 6, done.
# ... (push details)
# To https://github.com/username/my-first-project.git
# * [new branch] main -> main
# Future pushes (after -u flag is set)
git push
# Push specific branch
git push origin feature-branch
# 3. Pull updates from remote
git pull
# OR more explicit
git pull origin main
# Pull is equivalent to fetch + merge:
git fetch origin
git merge origin/main
# 4. Handle divergent branches
# If remote has changes you don't have:
git pull
# May result in merge commit
# Alternative: rebase instead of merge
git pull --rebase
# This replays your commits on top of remote changes
# View remote branches
git branch -r
# Output:
# origin/main
# origin/feature-branch
# View all branches (local and remote)
git branch -a
Common Remote Operations#
# Clone existing repository
git clone https://github.com/username/repo.git
cd repo
# Fetch without merging
git fetch origin
# See what would be pulled
git fetch
git log HEAD..origin/main
# Push force (dangerous! use with caution)
git push --force # Overwrites remote history
git push --force-with-lease # Safer: fails if someone else pushed
# Delete remote branch
git push origin --delete feature-branch
# Rename remote
git remote rename origin upstream
# Change remote URL
git remote set-url origin new-url
Exercise 4: Undoing Changes#
Tasks:
Undo changes in working directory
Unstage files
Amend last commit
Revert a commit
Solution#
# 1. Undo changes in working directory (not yet staged)
# Discard changes to specific file
git checkout -- file.txt
# OR newer syntax
git restore file.txt
# Discard all changes
git checkout -- .
# OR
git restore .
# 2. Unstage files (undo git add)
git reset HEAD file.txt
# OR newer syntax
git restore --staged file.txt
# Unstage all files
git reset HEAD
# OR
git restore --staged .
# 3. Amend last commit
# Fix commit message
git commit --amend -m "New commit message"
# Add forgotten file to last commit
git add forgotten-file.txt
git commit --amend --no-edit
# 4. Revert a commit (creates new commit)
# Revert last commit
git revert HEAD
# Revert specific commit
git revert abc1234
# Revert without committing (for editing)
git revert --no-commit abc1234
# Make changes, then:
git commit -m "Revert abc1234 with modifications"
Advanced Undo Operations#
# Reset to previous commit (dangerous!)
# Soft reset: Keep changes staged
git reset --soft HEAD~1
# Mixed reset: Keep changes unstaged (default)
git reset HEAD~1
# OR
git reset --mixed HEAD~1
# Hard reset: Discard all changes (DANGEROUS!)
git reset --hard HEAD~1
# Reset to specific commit
git reset --hard abc1234
# Undo reset (find lost commits)
git reflog
# Find the commit before reset
git reset --hard HEAD@{1}
# Clean untracked files
# See what would be deleted
git clean -n
# Delete untracked files
git clean -f
# Delete untracked files and directories
git clean -fd
Safety Tips#
# Before dangerous operations, create backup branch
git branch backup-$(date +%Y%m%d-%H%M%S)
# Check what you're about to do
git diff # Changes not staged
git diff --staged # Changes staged for commit
git diff HEAD # All changes
# Stash changes temporarily
git stash
# Do something else
git stash pop # Restore changes
Exercise 5: Advanced Git Workflows#
Tasks:
Interactive rebase
Cherry-pick commits
Git stash workflow
Tagging releases
Solution#
# 1. Interactive rebase (edit commit history)
# Rebase last 3 commits
git rebase -i HEAD~3
# Opens editor with:
# pick abc1234 Commit message 1
# pick def5678 Commit message 2
# pick ghi9012 Commit message 3
# You can:
# pick = keep commit
# reword = change commit message
# edit = stop to amend commit
# squash = combine with previous commit
# fixup = like squash but discard message
# drop = remove commit
# Example: Squash last 3 commits into one
# pick abc1234 Commit message 1
# squash def5678 Commit message 2
# squash ghi9012 Commit message 3
# Save and close editor, then edit final commit message
# 2. Cherry-pick commits
# Apply specific commit from another branch
git cherry-pick abc1234
# Cherry-pick multiple commits
git cherry-pick abc1234 def5678
# Cherry-pick without committing (for editing)
git cherry-pick --no-commit abc1234
# 3. Git stash workflow
# Save work in progress
git stash
# OR with message
git stash save "WIP: working on feature X"
# List stashes
git stash list
# Output:
# stash@{0}: WIP: working on feature X
# stash@{1}: WIP on main: abc1234 Commit message
# Apply latest stash (keeps it in stash list)
git stash apply
# Apply and remove from stash list
git stash pop
# Apply specific stash
git stash apply stash@{1}
# View stash contents
git stash show -p stash@{0}
# Delete stash
git stash drop stash@{0}
# Delete all stashes
git stash clear
# 4. Tagging releases
# Create lightweight tag
git tag v1.0.0
# Create annotated tag (recommended for releases)
git tag -a v1.0.0 -m "Version 1.0.0: First stable release"
# Tag specific commit
git tag -a v0.9.0 abc1234 -m "Version 0.9.0: Beta release"
# List tags
git tag
# OR with pattern
git tag -l "v1.*"
# Show tag details
git show v1.0.0
# Push tag to remote
git push origin v1.0.0
# Push all tags
git push --tags
# Delete local tag
git tag -d v1.0.0
# Delete remote tag
git push origin --delete v1.0.0
# Checkout specific tag
git checkout v1.0.0
# This puts you in "detached HEAD" state
Semantic Versioning#
Format: v{MAJOR}.{MINOR}.{PATCH}
Examples:
v1.0.0 - Major release
v1.1.0 - Minor update (new features, backward compatible)
v1.1.1 - Patch (bug fixes)
v2.0.0 - Breaking changes
Pre-release:
v1.0.0-alpha
v1.0.0-beta.1
v1.0.0-rc.1 (release candidate)
04 - Text Editors (Vim)#
Exercise 2: Vim Editing#
Tasks: Edit text efficiently
Solution#
# === ENTERING INSERT MODE ===
i # Insert before cursor
a # Append after cursor
I # Insert at start of line
A # Append at end of line
o # Open new line below
O # Open new line above
# Exit insert mode: ESC
# === EDITING COMMANDS (NORMAL MODE) ===
# Delete
x # Delete character
dw # Delete word
dd # Delete line
D # Delete to end of line
5dd # Delete 5 lines
# Change (delete and enter insert mode)
cw # Change word
cc # Change line
C # Change to end of line
ciw # Change inner word (from anywhere in word)
ci" # Change inside quotes
ci( # Change inside parentheses
# Copy (yank)
yw # Yank word
yy # Yank line
Y # Yank line (same as yy)
5yy # Yank 5 lines
# Paste
p # Paste after cursor/below line
P # Paste before cursor/above line
# Undo/Redo
u # Undo
Ctrl+r # Redo
# Repeat last command
. # Dot repeats last change
Exercise 3: Vim Visual Mode#
Tasks: Select and manipulate text blocks
Solution#
# === VISUAL MODE ===
v # Enter visual mode (character-wise)
V # Enter visual line mode
Ctrl+v # Enter visual block mode
# After entering visual mode:
# - Use movement keys to select text
# - Then use commands:
d # Delete selection
y # Yank (copy) selection
c # Change selection
> # Indent right
< # Indent left
= # Auto-indent
~ # Toggle case
u # Lowercase
U # Uppercase
# === PRACTICAL EXAMPLES ===
# Select entire file
ggVG
# Select paragraph
vip
# Select inside quotes
vi"
# Select inside brackets
vi{
# Visual block mode (column editing)
# 1. Place cursor at start
# 2. Ctrl+v
# 3. Move down/right to select block
# 4. I (capital i) to insert
# 5. Type text
# 6. ESC (changes apply to all lines)
Exercise 4: Vim Search and Replace#
Tasks: Find and replace text patterns
Solution#
# === SEARCH AND REPLACE ===
# Basic substitute command syntax:
# :s/pattern/replacement/flags
# Replace first occurrence on current line
:s/old/new/
# Replace all occurrences on current line
:s/old/new/g
# Replace all occurrences in entire file
:%s/old/new/g
# Replace with confirmation
:%s/old/new/gc
# y = yes, n = no, a = all, q = quit
# Replace in line range
:10,20s/old/new/g # Lines 10-20
:.,+10s/old/new/g # Current line + 10 lines
# Replace in visual selection
# 1. Select text with V
# 2. :'<,'>s/old/new/g
# (:'<,'> is automatically inserted)
# Case-insensitive search
:%s/old/new/gi
# Preserve case
:set ignorecase
:set smartcase
# === ADVANCED PATTERNS ===
# Delete empty lines
:g/^$/d
# Delete lines containing pattern
:g/pattern/d
# Keep only lines containing pattern
:v/pattern/d
# Replace with captured groups
:%s/\(\w\+\) \(\w\+\)/\2 \1/
# Swaps first two words on each line
# Multiple replacements
:%s/old1/new1/g | %s/old2/new2/g
Exercise 5: Vim Configuration#
Tasks: Customize Vim with ~/.vimrc
Solution#
# Create/edit .vimrc file
vim ~/.vimrc
Add this content:
" === BASIC SETTINGS ===
" Enable syntax highlighting
syntax on
" Show line numbers
set number
" Relative line numbers (easier for movements like "5j")
set relativenumber
" Enable mouse support
set mouse=a
" Show current mode
set showmode
set showcmd
" === INDENTATION ===
" Use spaces instead of tabs
set expandtab
" Tab width = 4 spaces
set tabstop=4
set shiftwidth=4
set softtabstop=4
" Auto-indent new lines
set autoindent
set smartindent
" === SEARCH ===
" Highlight search results
set hlsearch
" Incremental search
set incsearch
" Case-insensitive search
set ignorecase
" Unless uppercase is used
set smartcase
" === UI ===
" Show cursor position
set ruler
" Highlight current line
set cursorline
" Always show status line
set laststatus=2
" Command line height
set cmdheight=2
" === KEYBINDINGS ===
" Leader key
let mapleader = ","
" Clear search highlighting with <leader> + space
nnoremap <leader><space> :nohlsearch<CR>
" Quick save with <leader> + w
nnoremap <leader>w :w<CR>
" Quick quit with <leader> + q
nnoremap <leader>q :q<CR>
" Move lines up/down with Alt+j/k
nnoremap <A-j> :m .+1<CR>==
nnoremap <A-k> :m .-2<CR>==
vnoremap <A-j> :m '>+1<CR>gv=gv
vnoremap <A-k> :m '<-2<CR>gv=gv
" === MISC ===
" Enable file type detection
filetype plugin indent on
" Backup and swap file location
set backup
set backupdir=~/.vim/backup//
set directory=~/.vim/swap//
set undodir=~/.vim/undo//
" Create directories if they don't exist
if !isdirectory($HOME."/.vim/backup")
call mkdir($HOME."/.vim/backup", "p")
endif
if !isdirectory($HOME."/.vim/swap")
call mkdir($HOME."/.vim/swap", "p")
endif
if !isdirectory($HOME."/.vim/undo")
call mkdir($HOME."/.vim/undo", "p")
endif
" Auto-reload .vimrc when saved
autocmd BufWritePost .vimrc source %
Apply the configuration:
# Reload Vim or source the file
:source ~/.vimrc
# OR restart Vim
Quick Reference Card#
Mode Switches:
ESC → Normal mode
i/a/o/I/A/O → Insert mode
v/V/Ctrl+v → Visual mode
: → Command mode
Essential Commands:
:w - Save
:q - Quit
:wq - Save and quit
:q! - Quit without saving
u - Undo
Ctrl+r - Redo
/text - Search
n/N - Next/previous match
dd - Delete line
yy - Copy line
p - Paste
. - Repeat last change
06 - Debugging & Profiling#
Solutions for debugging and profiling exercises are best practiced hands-on with actual buggy code.
Key Debugging Techniques#
Using pdb (Python Debugger)#
# Insert breakpoint in code
import pdb; pdb.set_trace()
# OR Python 3.7+
breakpoint()
# pdb commands:
# n (next) - Execute next line
# s (step) - Step into function
# c (continue) - Continue execution
# l (list) - Show code context
# p variable - Print variable
# pp variable - Pretty print
# w (where) - Show stack trace
# q (quit) - Quit debugger
Performance Profiling#
# Profile with cProfile
python -m cProfile -s cumulative script.py
# In code:
import cProfile
cProfile.run('my_function()')
# Line profiler (requires line_profiler package)
@profile
def my_function():
# code here
pass
# Run with:
kernprof -l -v script.py
Memory Profiling#
# Using memory_profiler
from memory_profiler import profile
@profile
def my_function():
# code here
pass
# Run with:
python -m memory_profiler script.py
For complete solutions, refer to the original notebook 06_debugging_profiling.ipynb which contains interactive debugging exercises.
07 - Security Essentials#
Important: Legal and Ethical Use Only#
⚠️ WARNING: Security testing techniques should only be used:
On systems you own
With explicit written authorization
In controlled lab environments
For educational purposes
Unauthorized access is illegal.
Exercise 1: Password Security#
Task: Implement secure password hashing
Solution#
# NEVER store passwords in plain text!
# NEVER use MD5 or SHA1 for passwords!
# ALWAYS use bcrypt, Argon2, or scrypt
# Install: pip install bcrypt
import bcrypt
def hash_password(password: str) -> bytes:
"""
Hash a password using bcrypt.
bcrypt automatically:
- Generates a salt
- Uses key stretching (slow, expensive)
- Includes version and cost factor in hash
"""
# Encode password to bytes
password_bytes = password.encode('utf-8')
# Generate salt and hash
salt = bcrypt.gensalt(rounds=12) # Cost factor: 2^12 iterations
hashed = bcrypt.hashpw(password_bytes, salt)
return hashed
def verify_password(password: str, hashed: bytes) -> bool:
"""
Verify a password against its hash.
"""
password_bytes = password.encode('utf-8')
return bcrypt.checkpw(password_bytes, hashed)
# Example usage
password = "MySecurePassword123!"
# Hash password (do this once when user registers/changes password)
hashed = hash_password(password)
print(f"Hashed: {hashed}")
# Verify password (do this when user logs in)
is_valid = verify_password(password, hashed)
print(f"Password valid: {is_valid}")
# Wrong password
is_valid = verify_password("WrongPassword", hashed)
print(f"Wrong password valid: {is_valid}")
Password Best Practices#
# Password requirements (for user-facing systems)
def is_strong_password(password: str) -> bool:
"""
Check if password meets minimum requirements.
NIST guidelines (2017+):
- Minimum 8 characters (12+ recommended)
- No complexity requirements (allows passphrases)
- Check against breach databases
- No arbitrary restrictions (max length, special chars)
"""
if len(password) < 12:
return False
# Check against common passwords
common_passwords = {'password', '123456', 'admin', ...}
if password.lower() in common_passwords:
return False
return True
# API key generation
import secrets
def generate_api_key(length: int = 32) -> str:
"""
Generate cryptographically secure API key.
"""
return secrets.token_urlsafe(length)
# Example
api_key = generate_api_key()
print(f"API Key: {api_key}")
Exercise 2: Input Validation#
Task: Prevent injection attacks
Solution#
# SQL Injection Prevention
# ❌ VULNERABLE CODE
def login_vulnerable(username, password):
query = f"SELECT * FROM users WHERE username='{username}' AND password='{password}'"
# Attacker input: username = "admin' OR '1'='1"
# Results in: SELECT * FROM users WHERE username='admin' OR '1'='1' AND ...
# Always returns true!
# ✅ SECURE CODE - Use parameterized queries
import sqlite3
def login_secure(username, password):
conn = sqlite3.connect('database.db')
cursor = conn.cursor()
# Use ? placeholders (SQLite) or %s (MySQL)
query = "SELECT * FROM users WHERE username=? AND password=?"
cursor.execute(query, (username, password)) # Parameters passed separately!
result = cursor.fetchone()
conn.close()
return result
# XSS (Cross-Site Scripting) Prevention
# ❌ VULNERABLE CODE
def display_comment_vulnerable(comment):
return f"<p>{comment}</p>" # Directly inserts user input
# Attacker input: <script>alert('XSS')</script>
# ✅ SECURE CODE - Escape HTML
import html
def display_comment_secure(comment):
escaped = html.escape(comment) # Converts < to <, etc.
return f"<p>{escaped}</p>"
# Command Injection Prevention
# ❌ VULNERABLE CODE
import os
def ping_vulnerable(hostname):
os.system(f"ping -c 1 {hostname}") # Shell injection!
# Attacker input: "example.com; rm -rf /"
# ✅ SECURE CODE - Use subprocess with list
import subprocess
def ping_secure(hostname):
# Validate input first
if not hostname.replace('.', '').replace('-', '').isalnum():
raise ValueError("Invalid hostname")
# Use list arguments (no shell interpretation)
result = subprocess.run(
['ping', '-c', '1', hostname],
capture_output=True,
text=True,
timeout=5
)
return result.stdout
# Path Traversal Prevention
# ❌ VULNERABLE CODE
def read_file_vulnerable(filename):
with open(f"/var/www/uploads/{filename}") as f:
return f.read()
# Attacker input: "../../../etc/passwd"
# ✅ SECURE CODE - Validate and sanitize
import os
from pathlib import Path
def read_file_secure(filename):
# Define allowed directory
base_dir = Path("/var/www/uploads").resolve()
# Resolve requested file path
requested_path = (base_dir / filename).resolve()
# Check if path is within allowed directory
if not requested_path.is_relative_to(base_dir):
raise ValueError("Path traversal detected")
# Check file exists and is a file
if not requested_path.is_file():
raise FileNotFoundError("File not found")
with open(requested_path) as f:
return f.read()
Input Validation Framework#
import re
from typing import Any
class InputValidator:
"""General input validation framework."""
@staticmethod
def validate_email(email: str) -> bool:
pattern = r'^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$'
return bool(re.match(pattern, email))
@staticmethod
def validate_username(username: str) -> bool:
# Alphanumeric, dash, underscore, 3-20 characters
pattern = r'^[a-zA-Z0-9_-]{3,20}$'
return bool(re.match(pattern, username))
@staticmethod
def sanitize_string(text: str) -> str:
"""Remove potentially dangerous characters."""
# Allow only alphanumeric, spaces, and basic punctuation
return re.sub(r'[^a-zA-Z0-9\s.,!?-]', '', text)
@staticmethod
def validate_integer(value: Any, min_val: int = None, max_val: int = None) -> bool:
try:
num = int(value)
if min_val is not None and num < min_val:
return False
if max_val is not None and num > max_val:
return False
return True
except (ValueError, TypeError):
return False
# Usage
validator = InputValidator()
print(validator.validate_email("user@example.com")) # True
print(validator.validate_email("invalid-email")) # False
print(validator.validate_username("john_doe123")) # True
print(validator.validate_integer("42", 0, 100)) # True
Exercise 3: Secrets Management#
Task: Securely handle API keys and credentials
Solution#
# ❌ NEVER DO THIS
API_KEY = "sk_live_abc123xyz789" # Hardcoded secret
DATABASE_PASSWORD = "admin123" # Hardcoded password
# ✅ CORRECT APPROACHES
# 1. Environment Variables
import os
from dotenv import load_dotenv # pip install python-dotenv
# Load from .env file (NEVER commit .env to git!)
load_dotenv()
API_KEY = os.getenv('API_KEY')
DATABASE_URL = os.getenv('DATABASE_URL')
if not API_KEY:
raise ValueError("API_KEY environment variable not set")
# 2. Config file (also add to .gitignore)
import json
def load_config():
with open('config.json') as f: # Add config.json to .gitignore
return json.load(f)
config = load_config()
api_key = config['api_key']
# 3. Secrets management service (production)
# - AWS Secrets Manager
# - Azure Key Vault
# - HashiCorp Vault
# - Google Cloud Secret Manager
# Example with AWS Secrets Manager
import boto3
from botocore.exceptions import ClientError
def get_secret(secret_name):
session = boto3.session.Session()
client = session.client('secretsmanager')
try:
response = client.get_secret_value(SecretId=secret_name)
return json.loads(response['SecretString'])
except ClientError as e:
raise e
secrets = get_secret('myapp/prod/credentials')
api_key = secrets['api_key']
.env File Example#
Create .env file:
# .env - NEVER commit this file!
API_KEY=sk_live_abc123xyz789
DATABASE_URL=postgresql://user:pass@localhost:5432/dbname
SECRET_KEY=your-secret-key-here
DEBUG=False
Add to .gitignore:
# .gitignore
.env
config.json
secrets/
*.key
*.pem
Provide template:
# .env.example - COMMIT this file
API_KEY=your_api_key_here
DATABASE_URL=postgresql://user:pass@localhost:5432/dbname
SECRET_KEY=generate_random_secret_key
DEBUG=True
Security Checklist#
# Security checklist for your code
SECURITY_CHECKLIST = [
"✅ Passwords hashed with bcrypt/Argon2",
"✅ SQL queries use parameterized statements",
"✅ User input validated and sanitized",
"✅ HTML output escaped",
"✅ Secrets in environment variables, not code",
"✅ HTTPS used for all traffic",
"✅ Authentication required for sensitive operations",
"✅ Authorization checks before data access",
"✅ Rate limiting implemented",
"✅ Error messages don't leak sensitive info",
"✅ Dependencies regularly updated",
"✅ Security headers set (CSP, HSTS, etc.)",
"✅ Logging enabled (but not logging secrets!)",
"✅ Regular security audits",
]
Summary#
You’ve completed all Developer Tools exercises! You now have practical skills in:
Command Line Mastery#
Shell navigation and file management
Text processing with grep, sed, awk
Process management
Pipelines and automation
Version Control#
Git fundamentals (init, add, commit)
Branching and merging workflows
Remote repositories (push, pull, fetch)
Advanced Git (rebase, stash, tags)
Text Editing#
Vim modal editing
Efficient navigation and editing
Search and replace
Customization with .vimrc
Debugging & Security#
Debugging techniques (pdb)
Performance profiling
Secure coding practices
Input validation
Secrets management
Next Steps#
Practice daily: Use these tools in your regular workflow
Build projects: Apply skills to real repositories
Learn more tools: Docker, CI/CD, SSH, package managers
Contribute to open source: Real-world Git experience
Security mindset: Always think about security implications
Additional Resources#
Missing Semester (MIT): https://missing.csail.mit.edu/
Pro Git Book: https://git-scm.com/book
Vim Adventures: https://vim-adventures.com/
OWASP Top 10: https://owasp.org/www-project-top-ten/
Command Line Challenge: https://cmdchallenge.com/
Keep practicing, and these tools will become second nature! 🚀