Mirza Ishraq

How to Get the Most Out of a $20 Claude Plan as a Developer

Wed, 23 Sep 2026 00:00:00 GMT

The $20 Claude plan is Pro, and it includes Claude Code. Claude Code on Pro now defaults to Opus 5.5 at medium effort, and with the September 22 launch Anthropic said it is raising five-hour limits on the same plans.

You get the most out of it by spending fewer tokens per task, not by using Claude less. That means one clear message instead of five, a fresh context between tasks, the right model and effort for the job, and no cold-cache surprises after a break.

One thing up front. I started on Pro, brand new to Claude Code, and I’m on the $200 Max plan now. The prices and limits below come from Anthropic’s pages, and my usage numbers come from Max.

Every factual claim about Anthropic’s plans and tools links to the page it came from, and I checked those pages on September 23, 2026. Sonnet 5.5 and Haiku 5.5 are due “in the coming weeks,” so parts of this will age. Anthropic doesn’t publish Pro token allowances, and I won’t invent one.

Here is the scorecard before the detail.

Habit	Why it helps on a $20 plan
Know where your config lives	Settings and `CLAUDE.md` files do nothing if they are in the wrong place.
One message: task, finish line, stop rule	Fewer turns, and every turn re-reads the whole conversation.
`/clear` between tasks	It costs nothing and drops stale context.
Start effort at `medium`, then experiment	You can’t turn thinking off on Opus 5.5. Effort is the dial.
`opusplan`	Opus 5.5 plans, Sonnet 5 types.
Skip Fable, fast mode and the advisor	Each one bills more or reads the transcript uncached.
Use ultracode for one job, not all day	It applies to every task in the session and fans out into many agents.
Mind the one-hour cache	The first message after a long break re-reads everything.

What does a $20 Claude plan give a developer?

Pro costs $20 a month, or $17 a month if you pay $200 up front, and it includes Claude Code. The plan also covers Claude Code inside VS Code, Cursor and JetBrains IDEs, per Anthropic’s Help Center.

Claude apps and Claude Code draw from one shared pool. Limits reset on a rolling five-hour session window, and there is a weekly limit that applies across all models. A long chat in the app and a long run in the terminal come out of the same tank.

The default model changed recently. The model docs say Pro now defaults to Opus 5.5, with medium as its default effort. Before Claude Code v2.1.280, Pro defaulted to Sonnet 5, so on an older version you may still see Sonnet.

The Opus 5.5 announcement adds one line: “we’re increasing five-hour usage limits on Pro, Max, Team, and seat-based Enterprise plans.” It gives no numbers.

Several blog posts quote “about 45 prompts” or “44,000 tokens” per window. I couldn’t find either in the Anthropic pages I read, which describe Pro only as “more usage” than Free. I’m leaving both out. Treat any exact Pro number as a guess unless it links to Anthropic.

What did I get wrong on my first Pro plan?

I started on Pro, and I was new to Claude Code. I didn’t know the tweaks or the ins and outs: the global config, the .claude folder, settings.json, CLAUDE.md. I wasn’t following Anthropic’s guidelines either, so I was barely able to use my $20 plan.

When I hit the limit, I couldn’t ask Claude anything else, so I reviewed the code it had generated by hand. Those were my early days of AI-assisted coding, and I took help from ChatGPT.

If that sounds familiar, here is the map I didn’t have. This is where Claude Code’s settings and memory files live:

~/.claude/settings.json is your user settings, for you in every project. Claude Code creates it the first time you change an option in /config that it stores there, such as the theme.
.claude/settings.json inside a repo is the shared project file, for everyone who clones it.
.claude/settings.local.json is your personal override for one project. Claude Code keeps it out of git when it creates the file. If you make it by hand, add it to .gitignore yourself.
~/.claude/CLAUDE.md holds your personal instructions for every project.
./CLAUDE.md or ./.claude/CLAUDE.md holds the project’s instructions, shared with your team through version control.

When the same key shows up in more than one settings file, the higher level wins. Project local beats shared project, and both beat your user settings.

Two traps I’d have avoided. First, CLAUDE.md is context, not enforcement. Claude treats it “as context, not enforced configuration,” so to block an action for certain, use a PreToolUse hook.

Second, keep it short. The size guidance says: “Longer files consume more context and reduce adherence.” Aim for under 200 lines, and run /context to see which memory files loaded.

The rest of this post is the guideline pass I skipped.

What actually eats your Claude plan limits?

Context, mostly. Claude Code sends your full conversation with every request. So “a one-line question in a session that has been open all day still draws usage for the whole conversation.”

Every tool call is another request carrying the same history. That is why one wandering session costs more than three short ones. I broke down the numbers in my Claude Code cost post, including why cache reads make the re-read cheap but not free.

Anthropic’s usage limit guide lists several other inputs, including:

Message length
Attached file size
Conversation length
Tool use
Model choice
Effort level

So the levers are what you send, how long the chat runs, and which model and effort reads it.

Then there is the cache clock. On a subscription, the main conversation’s cache lives for one hour while you are inside plan usage. Come back after a longer break and the first message reprocesses everything.

There are two catches. Subagents get a five-minute cache even on a subscription. And once you pass your limit and draw on usage credits, the main conversation drops to five minutes too.

Background work counts as well. A /loop or scheduled task fires on its interval even while you are away, and it sends your full context each time.

One more fact. You can’t turn thinking off on Opus 5.5, and thinking tokens are billed as output tokens.

So effort is the only dial you get. On a plan, /usage shows what your limits went to and flags long context or cache misses once one reaches 10 percent of recent use. Run it before you change anything.

You can also see the five-hour and weekly bars under Settings > Usage on claude.ai.

Should you change the effort level on Opus 5.5?

Start at medium, then play with it. In the model docs, medium “reduces token usage for cost-sensitive work that can trade off some intelligence,” and low is for “short, scoped, latency-sensitive tasks that are not intelligence-sensitive.”

Anthropic’s own Opus 5.5 playbook, written by Addy Osmani, has one useful data point. “One early tester said Opus 5.5 at its lowest effort caught more bugs than Opus 5 at high effort, with fewer false alarms.”

One tester is an anecdote, not a benchmark. It is still a fair reason to try low on scoped work like a review or a rename.

The right level depends on your work, so try low, medium and high on the same kind of task and read /usage afterward. Test max before adopting it broadly. It “may show diminishing returns and is prone to overthinking.”

Switching is cheap on this model. On Opus 5.5 with a subscription, changing effort mid-session keeps the cache. On most other models it doesn’t.

Now a mismatch I found in my own settings. My user settings have "effortLevel": "high". I’m on a Max plan, and that isn’t a recommendation for a $20 plan.

The model docs say a top-level effortLevel in your user settings “doesn’t count for Opus 5.5.” Opus 5.5 should start at medium until you pick a level with /effort or the /model picker.

My own session log disagrees. On Claude Code 2.1.281, 52 Opus 5.5 turns in one session recorded high, both as the effort and as a per-turn effort. /effort also shows high in my session. That matches what my settings file says, not what the docs describe.

I can’t tell whether the docs lag the software or something else set the level, and the log format isn’t documented. So don’t take my word or the docs’ word. If your default is high on a $20 plan, Opus 5.5 may be running above its medium default and using more of your limit.

Under opusplan, /effort shows the model you are on right now, so check it in plan mode to see Opus 5.5’s level. Here is how to see what your session really uses:

Run /effort with no arguments. It opens an interactive slider. Press Enter to save a level as your default, or s to apply it to this session only.
Read the session header. It shows the level next to the model name, for example “with low effort,” and the footer briefly shows it at startup and when it changes.
Use a status line. Its effort.level field reports the live level. Ultracode is not a distinct level there and reports as xhigh.
Clear a saved level. /effort auto removes the level you saved for the active model.

Claude desktop reads the same settings files as the terminal, so the same effortLevel line applies there. I couldn’t find where the desktop docs put the effort control. Mine currently shows ultracode, which is a Claude Code setting that sends xhigh plus workflows, not a level of its own.

To pin a level for one model, use modelSettings:

{
  "modelSettings": {
    "claude-opus-5-5": { "effortLevel": "medium" }
  }
}

Change the value when /usage gives you a reason.

There is one more cheap win. The playbook says Opus 5.5 “always thinks before it replies, and it decides how much,” so “think carefully” lines are dead weight.

Anthropic tested this in a chat product. Removing one “made replies start sooner, with no clear drop in quality.” I searched my project CLAUDE.md and my /wrap and write-post skills for lines like that and found none. Yours might have some.

Which model should do which job?

Pick the cheapest one that can do the job. Anthropic’s Help Center says Sonnet suits the large majority of coding work. Opus is for harder problems like large refactors, and it uses more quota. Haiku is for quick lookups and simple edits.

For a rough sense of the gap, look at API list prices: $4 in and $20 out per million tokens for Opus 5.5, against $2 and $10 for Sonnet 5. That is API pricing, not plan quota. Read it as a ratio of two to one, not a promise.

That is the logic behind opusplan. It runs Opus 5.5 in plan mode and Sonnet 5 for execution.

My usage shows the split working. On September 23 so far, ccusage put Sonnet 5 at about 52 percent of my tokens and Opus 5.5 at about 30 percent. That is a Max account and list-price accounting, so it shows the split, not what Pro would allow.

One catch, covered in my cost post. Each plan-mode toggle is a model switch, and each model has its own cache. So opusplan is cheapest when the context is small at the switch.

How has Opus 5.5 felt so far?

Since launch, it has felt faster and more focused. Anthropic says it generates output more than 30 percent faster than Opus 5, so that fits, though I didn’t time anything.

I also haven’t drained my limits, even after some heavy testing. My usage screen right now shows the current session at 12 percent and the weekly limit at 71 percent, with the reset on Friday morning. So it is not untouched, and a heavy week does show up.

For scale, ccusage counts 5.76 billion tokens on my account since Friday, September 18, which is roughly the same window as that weekly bar. About 97 percent are cache reads, meaning conversations being re-read. Fresh input and output add up to about 1 percent: around 37 million input and 22.5 million output tokens.

I can’t turn any of that into a Pro number. I found nothing from Anthropic on how tokens map to the bar, the bar also counts claude.ai, and I have no figures from my Pro days.

Read all of that with two caveats. I’m on Max, which gives at least 5x Pro’s usage. And most of my work stayed on Sonnet 5 through opusplan, so this was not an Opus-only test.

I’m not on Pro now, so I can’t tell you how Opus 5.5 feels on $20. That is why I’d watch /usage in your first week instead of trusting my impression.

How do you hand Opus 5.5 a task without wasting turns?

Give it the whole task in one message. Anthropic’s playbook says to name what “done” looks like and when you want it to stop and ask, then let it work. That works best when you already know what you want built, which is the gap between AI-assisted coding and vibe coding.

My read on why that saves usage: every extra turn re-sends your whole conversation. Five small messages are five full reads. Here is the shape, with a made-up project:

Add rate limiting to the login route.
Done means: 5 attempts per minute per IP, the new tests pass, and
npm run check is clean.
Stop and ask me only if a test fails and you can't explain why.

Three more habits from the playbook save turns too.

Type a follow-up while it runs. If you remember something mid-run, type it and press Enter instead of restarting. In the playbook’s words, “Runs are longer now, so a restart costs more.”

Keep the task list in a file. Ask Claude to keep a checklist in TASKS.md. A long run fills the context window, Claude Code then summarizes older turns, and a list in a file survives that.

Tell it which stops you want. Opus 5.5 sometimes stops to report or asks “Want me to continue?” Each stop costs you another turn, and every turn re-reads the context.

A short rule in CLAUDE.md fixes most of it. This one is adapted from the playbook:

When a step doesn't need my input, keep going. Put status notes in the
same message as your next action.
Stop and ask only when you can't continue without me, or before
anything destructive: deleting data, force-pushing, or changing
anything outside this repository.

Keep permission prompts on for destructive commands anyway. Claude Code auto mode adds a second wall: a classifier that reviews risky actions before they run.

Two habits from the Help Center are worth adding. Point Claude at files by path instead of pasting them, and ask for a plan before a big change. A plan is cheap, and a wrong multi-file diff is not.

When should you clear, compact or rewind?

Clear between unrelated tasks. The costs docs put it plainly: “When you want a fresh start instead of continuity, /clear costs nothing.” Run /rename first if you want to /resume the session later.

Compact at a natural break, not in the middle of a task. Compaction is itself a request that reads the whole conversation. After a break longer than the cache lifetime, it reprocesses all of it uncached.

To abandon a path, use /rewind. It truncates back to a prefix that is already cached instead of building a new one the way compaction does.

Back from a long break? On Pro and Max, Claude Code offers to resume from a summary, so later requests don’t carry the full history. Say yes unless you need the detail.

Keep the always-loaded stuff small too. Anthropic suggests a CLAUDE.md under 200 lines, with workflow instructions moved into skills that load only when invoked.

Run /mcp and disable servers you aren’t using. I measured what those cost in the cost post. One gotcha: editing CLAUDE.md mid-session doesn’t apply until /clear, /compact or a restart.

Which Claude Code features should a $20 plan skip?

Four of them cost more than they look.

Fable. It isn’t the default on any plan. On some plans it bills to usage credits, and the /model picker shows “Requires usage credits” on its row. On my Max plan, the usage screen shows a separate weekly limit for Fable instead. The docs I read don’t say which case Pro is in, so check the picker before you select it.

Fast mode. The playbook says it “needs extra usage turned on, and it costs more per token than standard mode.” Turning it on mid-session also causes a cache miss. API list price is $8 in and $40 out per million tokens, against $4 and $20 in standard mode. If you use it, turn it on at the start of a session.

The advisor. Each call re-reads the full transcript with no caching, and on subscriptions the advisor docs say advisor usage counts toward your plan limits. I keep advisorModel on Opus, but that is a quality purchase on a Max plan. On $20, /advisor off is the first thing to try if you are short on usage.

Agent teams. Agent teams use about seven times the tokens of a standard session when teammates run in plan mode. They are off by default. Leave them off.

Should a $20 plan use ultracode?

Not as a default. Ultracode combines xhigh effort with automatic workflows, so Claude plans a workflow for each substantive task. That applies to every task in the session, “so each request uses more tokens and takes longer than at lower effort levels.” The docs also say to drop back with /effort high for routine work.

The cost is the fan-out. A workflow spawns many agents, so a single run can use meaningfully more tokens than doing the same task in conversation, and runs count toward your plan’s usage and rate limits.

Anthropic’s own post on dynamic workflows says they “often use more tokens and are best suited for complex, high value tasks.” It adds that most traditional coding tasks “do not need a panel of 5 reviewers.”

I tried ultracode on Opus 5.5 in Claude desktop for a large code review, which is the kind of job the docs use as an example. That was on Max, so it is a look at the feature, not a Pro test. The desktop app draws on the same plan usage as the other Claude Code surfaces, and its approval card carries a token-usage caution.

If you review often, a repeatable pipeline is another route. Mine is in my automated code review pipeline post.

Three details matter more on $20:

You turn workflows on yourself. On Pro, use the Dynamic workflows row in /config.
The size guideline defaults to small. On Pro with Claude Code v2.1.271 or later, Claude aims for fewer than five agents.
Ultracode hides the Large workflow warning. That warning normally appears past 25 agents or 1.5 million projected tokens. With ultracode on, you don’t see it, because turning it on “already opts you in to large runs.”

If you want to try one, there is a cheaper route:

Ask for one workflow, not a whole session. Put the keyword ultracode in a single prompt. It runs one task as a workflow without changing the session’s effort level.
Start on a slice. Try one directory instead of the whole repo, and watch each agent’s token use in /workflows. You can stop the run there.
Keep the size small. Run /config workflowSizeGuideline=small if yours isn’t already.
Know what a limit does. On v2.1.271 or later, a run pauses at your five-hour limit and continues after the reset, instead of failing. It only pauses if the limit resets within 24 hours, so a weekly limit can still fail the agents. The workflows docs list every condition.

What do you do when you hit the limit?

You have four options.

Wait. The limit message shows the reset time. On Claude Code v2.1.234 or later, it can also wait and pick the task back up after the reset.
Turn on usage credits. Pro is eligible. Usage is billed at standard API rates, and you set a monthly spending cap. The Help Center has the details. Remember the cache drops to five minutes once you are on credits.
Switch model families. If you see a model-specific message like “You’ve hit your Opus limit,” switching to a model outside that family with /model keeps you working. That guidance sits in the costs docs section for seat-based plans, so check it against your own message. A session or weekly limit is shared across models, so switching won’t fix that one.
Move up a tier. Max starts at $100 and gives you “5x or 20x more usage than Pro.”

Ask for the review while you still have usage left. I once had to review generated code by hand because I had already hit the limit.

If you hit the session limit most days even with these habits, that is not you doing it wrong. It is a sign Pro may be too small for how you work, and the point where Max becomes worth pricing out.

The settings I’d start with

Here is the small version. I run it on Max, not Pro, so treat it as a starting point.

{
  "model": "opusplan",
  "env": {
    "CLAUDE_CODE_SUBAGENT_MODEL": "haiku"
  }
}

Add the CLAUDE.md stop rule from above and delete any “think carefully” lines. The built-in Explore and Plan subagents ignore the variable on its own, as I found in the cost post.

Then measure. Run /usage and read the Prompt cache (main) line for your hit rate and misses. If a habit doesn’t move a number there, it is just a habit.

The short version: How to get the most out of your $20 plan

Here is the whole post as a checklist.

Know where your config lives. ~/.claude/settings.json is for you everywhere, .claude/settings.json is for the project, and CLAUDE.md should stay under about 200 lines.
Read /usage first. Check the Prompt cache (main) line so you know where your limits actually go.
Start effort at medium, then experiment. Try low on scoped jobs and high on hard ones, check /effort for what you are really on, pin levels with modelSettings, and delete “think carefully” lines from your prompts, CLAUDE.md and skills.
Use opusplan. Opus 5.5 plans and Sonnet 5 types.
Send the whole task in one message. Say what “done” means and when to stop, keep a checklist in TASKS.md for long runs, and type follow-ups mid-run instead of restarting.
Add the stop rule to CLAUDE.md. Keep going when a step needs no input, ask before anything destructive, and keep permission prompts on.
Keep context small. /clear between tasks, /compact at natural breaks, /rewind to abandon a path, and /mcp to turn off servers you don’t use.
Accept the resume-from-summary offer. Do it after a break longer than an hour, when the cache has gone cold.
Skip the expensive features. That means Fable, fast mode and agent teams, and /advisor off is worth trying too.
Use ultracode for one big job, not all day. Put the keyword in a single prompt, start on a small slice, and drop back with /effort high.
At the limit, pick a route. Wait for the reset, turn on usage credits with a spending cap, or price out Max if it happens most days.

Which of these would save you the most on a $20 Claude plan, and have you checked it against your own /usage lately?

Automated Code Review Pipeline with Claude Code

Tue, 22 Sep 2026 00:00:00 GMT

Automated code review doesn’t work by asking the same AI that wrote a change to also grade it. That’s not review, that’s the same reasoning agreeing with itself.

My own pipeline for Claude Code runs eight independent stages before anything merges: a different model, a fresh context with zero memory of why any line was written the way it was, or a real running system that can’t be talked out of its own behavior.

It didn’t come from one clean rewrite. It grew stage by stage, over months, every time a single pass let something through that a second one would have caught.

None of the stages get final authority on their own. Each one produces a verdict, a human reads it, and merge approval is a separate step from everything else passing, on purpose, even on a Tuesday when everything is green.

That human read isn’t a formality either. I read the actual diff myself, not just the review verdicts, and if something doesn’t sit right, I ask Claude to explain it directly before I’ll approve anything.

An explanation that doesn’t hold up under a direct question is its own finding, even when nothing upstream flagged it.

What automated code review with Claude Code actually runs

Here’s the full chain, in order.

Stage	What runs	What it checks
1. Plan	A capable model, reading real source	File-and-line-level plan, not a description of intent
2. Adversarial plan review	A fresh context, read-only tools	Every load-bearing assumption in the plan, against the actual files
3. Implement	Whatever model is fast enough	Follows the reviewed plan
4. Pre-push code review	Independent model, fresh context	The diff, cold, no memory of the implementation
5. CI	GitHub Actions	Test suite, real build, smoke test against the built artifact
6. Holistic PR review	Independent model, fresh context	Diff + description + CI, checked against each other
7. Functional test	A real running instance	The behavior, exercised by hand, not just asserted
8. Human approval	Me	Explicit, every time, never inferred

Stages 2, 4, and 6 are real Claude Code subagents on my machine, not a metaphor. plan-adversary runs before I exit plan mode, on a strong model, and its entire job is to refuse to believe the plan.

Its own rule, verbatim from its prompt: “every factual claim in the plan is a hypothesis until you open the file.” It won’t accept “this is already verified” or “nothing else uses this” without grepping for every consumer itself, and it separates what it confirmed from what it merely couldn’t check, because letting “unverifiable” quietly become “confirmed” is exactly how a stale plan gets implemented.

code-reviewer runs before every commit and push, on a fixed strong model, with a fresh context that has never seen the implementation decisions or the rationalizations that came with them. It’s the same model split I use for the rest of the day’s work too, a capable model reserved for judgment, a faster one for typing, which I broke down properly in Claude Code cost.

pr-reviewer runs once a PR is open, and its job is different in scope: not “is this diff correct,” but “is this the right, complete, honestly described unit of work to merge.” It checks whether the PR description’s “what changed” actually matches the diff, and whether the “tested” claims were actually run or just judged plausible.

There’s a fourth agent, docs-librarian, that isn’t one of the eight stages. It triggers whenever a change touches a service that talks to an external API, and its only job is checking the implementation against the actual current documentation instead of trusting anyone’s memory of how that API works, including the model’s own.

None of this runs through a hook. The instruction is just: run these yourself, every time, unasked, even on the smallest edit. No hooks, no markers, just do it.

That’s a deliberate choice and not a shortcut I haven’t gotten around to. A hook fires on a file pattern; it doesn’t know a one-line change to an auth check deserves the full chain while a copy edit doesn’t. An instruction that’s expected to be followed every time, unprompted, handles that judgment call in a way a static trigger can’t.

It’s the same bet I made writing about Claude Code auto mode: don’t beg the model with a prompt, build something that either runs or doesn’t. Here the “something” is a standing instruction treated as non-negotiable, not a classifier, but the reasoning is the same. Begging doesn’t scale, structure does.

Any check that gates something also has to fail in the right direction. A permission check, a classifier, a review stage that can’t reach a verdict should default to deny, not allow. A check that returns “nothing is restricted” when it can’t read its input is a vulnerability waiting for exactly that failure to happen in production, and it’s worth testing the error path on purpose, not just the happy path where everything loads correctly.

Stage 7 gets the same independence rule, just applied to people instead of models. I’m not the one who functional-tests a change before it ships.

Someone who has never seen the implementation clicks through it instead. They don’t know which code path was touched, so they can’t unconsciously avoid the spot that’s actually fragile, the same reason a fresh-context code review catches things a self-review doesn’t.

Familiarity with your own reasoning is exactly what makes you bad at finding the gaps in it.

What it actually catches

The categories that keep showing up: a regex that matches more, or less, than it was supposed to. A race condition where two requests hit the same check-then-act sequence against Redis or Postgres, so the second one clobbers a state the first one assumed was still true. A cache key that doesn’t capture everything that actually affects the cached value, so a stale or cross-context result gets served back as current.

None of those fail a build. They fail under a timing pattern or an input shape a single read-through doesn’t reproduce, which is exactly why a fresh-context review pass matters more than a more careful first pass would.

The person, or the model, who wrote the regex already believes it’s correct. It takes a reviewer with no stake in that belief to ask what it matches when the input isn’t the one it was tested against.

The technique that actually finds these is boring: grep for every consumer of whatever’s changing, not just the one the plan or the diff talks about.

A plan almost always describes the happy-path caller. The break is nearly always in the variant it didn’t mention, the retry path, the second client, the call site that passes an option nobody else does.

Reading the described change carefully doesn’t surface that. Searching for who else touches the same thing does.

Findings also get graded, not just listed: blocker, should-fix, or question, the same way across every stage.

A blocker is something that can cause wrong results, lost data, or a security hole. A should-fix is a real defect that isn’t a merge risk. A question is something that looks suspicious but can’t be confirmed from the code alone, so it’s phrased as one instead of asserted as fact.

Grading turns a review into a triage list instead of a wall of text, and it stops a genuinely dangerous finding from getting lost between ten nitpicks about naming.

The one rule I don’t break: docs over prompting

I don’t try to control the model through clever prompting. I lean on what’s actually documented and on structured output instead.

That’s the whole reason docs-librarian exists. Prompting an agent to “be careful with the Stripe webhook signature” is a suggestion the model can rationalize past. Pointing it at Stripe’s actual current webhook documentation, and having it check the implementation line against that page, isn’t a suggestion, it’s a fact the model either matches or doesn’t.

The failure mode I’m avoiding isn’t the model being careless. It’s the model being confidently wrong from memory about an API that shipped a breaking change since its training data, and no amount of “please double-check” in a prompt fixes that. Only the current page does.

It checks two layers, in order. First, whatever pinned reference docs already exist for that integration in the repo itself, because that’s the contract the code was actually built against, and a mismatch between the code and that pinned file is a real finding even when the live API would also accept what the code currently does.

Then it goes to the live documentation for anything the pinned reference doesn’t cover, and it says plainly when no pinned reference existed to check against, rather than quietly filling the gap from memory.

When the pinned file and the live docs disagree with each other, it reports both instead of picking one silently, because that disagreement is itself information: either the pin is stale, or the code drifted from a deliberate decision, and only a human reading both knows which.

The same distrust of memory applies to security rules that get computed rather than declared. If “is this restricted” is derived by parsing something else, a naming convention, a list extracted from documentation prose, instead of being an explicit rule, a well-intentioned edit to that other artifact can silently change what’s protected with nothing failing to signal it. So a derived rule like that gets a test that pins the expected result directly, independent of the artifact it’s derived from.

The CI stage: test suite, smoke test, Docker or Ubuntu, and when the minutes run out

Stage 5 isn’t one job. On GitHub Actions it’s a test suite, a real build of the deployable artifact, and at least one smoke test against that built artifact rather than against source running in a dev harness, because a test suite can pass while the packaging step silently drops a file the app needs at runtime.

Whether that build is a Docker build or a plain Ubuntu build depends on the project: a container image if that’s what actually ships, a bare Ubuntu runner if it doesn’t. Same principle either way, only the artifact changes.

GitHub’s own free tier gives a private repo 2,000 Actions minutes a month on GitHub-hosted runners. Past that, I move the project to a self-hosted runner instead of paying per-minute overage. Self-hosted runners are free to use with GitHub Actions, the cost is just whatever machine you point at the job, so it’s a straight swap once a project’s usage pattern makes that machine cheaper than the overage.

I run this same shape of pipeline on Tradion, the AI trading research platform I built, and on other projects I can’t name here. The stages don’t change with the project. What changes is only the build target in stage 5.

The four agents, generic versions

These are stripped-down, generic versions of the four agents, not the verbatim prompts running on my own projects. Swap in your own stack’s file paths, frameworks, and conventions; the structure and the rules are what matter, not the specifics.

---
name: plan-adversary
description: Reviews a proposed plan against the actual codebase before any
  code is written. Never writes code itself.
tools: Read, Grep, Glob, Bash
model: opus
---
Every factual claim in the plan is a hypothesis until you open the file.
Never accept, without checking: a quoted line number or function name,
"this is already the case," "nothing else uses this," or a claim that a
change is safe. Open every file the plan touches. Grep for every consumer
of what's changing, not just the one the plan describes. Record each
assumption as confirmed, false, or unverified, and never let unverified
quietly become confirmed.
Output: WILL BREAK, CLAIMS THAT ARE FALSE, UNVERIFIED, NOT ACCOUNTED FOR,
CORRECT AS WRITTEN. Verdict: safe to implement / safe with changes / do
not implement as written.

---
name: code-reviewer
description: Independently reviews a diff after implementation, before
  push. Fresh context, no memory of implementation decisions.
tools: Read, Grep, Glob, Bash
model: opus
---
You have no memory of why any line was written the way it was. Review the
diff cold. Check correctness, access control, injection risks, race
conditions and check-then-act patterns, and regressions against existing
tests. Every finding needs file:line, the exact code, a concrete failure
scenario, and a fix. Grade each: BLOCKER, SHOULD-FIX, QUESTION. If nothing
is wrong, say so plainly; don't manufacture findings to look thorough.

---
name: docs-librarian
description: Verifies integrations against real, current documentation
  instead of trusting model memory.
tools: Read, Grep, WebFetch, WebSearch
model: sonnet
---
Check any pinned reference docs in the repo first, that's the contract the
code was built against. Then check live documentation for anything not
covered. If the two disagree, report both, don't silently prefer one.
Never rely on memory for version-specific API details. Quote directly from
documentation when flagging an issue, with a direct URL for every one.

---
name: pr-reviewer
description: Reviews an opened pull request holistically, diff,
  description, and CI, before merge. Fresh context.
tools: Read, Grep, Glob, Bash, WebFetch
model: opus
---
Check CI status first; don't review behind a red check. Compare the PR
description against the actual diff: does "what changed" match, were the
"tested" claims actually run or just judged plausible? Review the
implementation for correctness, security, and regressions. Verdict: ready
to approve / ready to approve with nits / request changes / blocked.
Return the verdict, never post, merge, or approve anything yourself.

What it costs, and when I skip it

The full gate, all eight stages, takes about 25 to 30 minutes end to end on a typical change. That’s real time added to every merge, and it’s not free in tokens either.

Running a pipeline this heavy only makes sense if I’m not also leaking tokens everywhere else. I keep Claude Code itself tightly configured for cost outside the pipeline, the model splits, the context hygiene, the settings I actually run are in how tightly I keep Claude Code’s day-to-day cost down. Spending deliberately in one place and carelessly everywhere else isn’t a strategy, it’s just a bigger bill with extra steps.

So I don’t run it on everything. Pure content or docs changes, or a single-property config or style tweak, skip the plan-review and pre-push review stages entirely and go straight to CI plus the holistic review. A small follow-up fixing something a reviewer already flagged gets the same treatment.

Anything touching permissions, shared state, money, or data deletion, or anything else expensive to undo, runs the full eight stages, no shortcuts, no matter how small the diff looks.

The sizing rule isn’t “is the diff small.” It’s “could this plausibly cause a wrong behavior or a security issue.”

A one-line diff that flips an authorization check is small and gets the full chain. A two-hundred-line copy change gets none of it.

Honestly, I don’t always get the sizing right. There are days a three-line fix goes through the full 25 to 30 minutes because I didn’t stop to ask whether it needed to, and it’s hard to tell in the moment whether that’s discipline or just not trusting my own judgment about what counts as small.

I also don’t think this pipeline is the only good way to do this. It’s the shape that’s worked for me so far, not a claim that it’s optimal.

I’ve mistaken process for progress before: 347 commits didn’t save the open source AI project that never found a user, and a review pipeline this elaborate deserves the same question I didn’t ask early enough there, whether it’s actually earning its cost or just feels like it is.

I’m genuinely curious what other people’s version of this looks like, and what a leaner or smarter setup would catch that mine doesn’t. What does your own review pipeline actually look like?

I Despise Vibe Coding. Here's the Only Place It Belongs

Tue, 22 Sep 2026 00:00:00 GMT

I use AI-assisted coding every day. Claude Code is part of how I build everything. And I despise vibe coding.

That sounds like a contradiction. It isn’t. AI-assisted coding and vibe coding are not the same thing, and the difference is the whole point of this post.

So what is vibe coding?

Vibe coding is when you don’t read the code. You trust the AI to write everything. You accept whatever it spits out. Something breaks? Paste the error back in and hope for the best.

That’s not me being harsh. That’s literally how it started. Andrej Karpathy coined “vibe coding” in a post on X on February 2, 2025 (X blocks some readers, so here’s a readable copy of the same post). In that same post, he admitted that the code grows beyond his usual comprehension, that when the AI can’t fix a bug he just asks for random changes until it goes away, and that it’s not too bad for throwaway weekend projects.

Throwaway weekend projects. The guy who invented the term said that. Somewhere along the way, people skipped that part and started shipping vibe-coded apps to real users.

I tried it. Here’s what happened.

I’m not writing this from the sidelines. I’ve vibe coded, mostly in Google AI Studio, building things like Crumbs, a business manager for cottage bakers, and ClaimGuard AI, an insurance-claims assistant. At first it felt amazing. Type a sentence, get an app.

Then I actually looked at what I had.

The designs were AI slop: the same sloppy blue-purple gradient and generic rounded borders every AI app seems to ship with. Simple things were written in weirdly complicated ways. Stuff that should’ve been ten lines was fifty. And Gemini hallucinated data right inside the app. I couldn’t fix it, because I had no idea how the code that was supposed to catch it actually worked.

That bug is what stopped me. I realized I didn’t own what I built. The AI did. And the AI doesn’t care if it breaks.

The only place vibe coding belongs

Prototypes. Demos. Testing an idea on a weekend. Anything with no real users, no real data, and nothing that hurts anyone if it falls apart.

That’s it. That’s the list.

I’ve been trading for over 9 years, and vibe coding reminds me of trading on a hot tip with no plan. Sometimes it works. You feel like a genius. But you don’t know why it worked, so you can’t repeat it, and you definitely can’t protect yourself when it goes wrong. Fine with play money. Terrible with real money.

Why I despise vibe coding in production

AI builds stuff you don’t need. It over-engineers everything. You ask for one simple feature and it adds extra layers, extra config, and code for problems you don’t even have. It pulls you toward features you never planned. If you’re not reading, you don’t notice, and your app slowly fills up with code nobody asked for and nobody understands.

It produces slop. AI slop apps. AI slop designs. They look done in a screenshot and fall apart the second a real user does something unexpected.

Even the best models need steering. In my experience, Sonnet still needs a lot more hands-on direction than Opus, which is part of why I split the two by task instead of using one for everything. Even Opus over-engineers things I never asked for often enough that I’ve stopped being surprised by it.

No model gets it right every time. More than once I’ve asked it to fix a bug and gotten a list of two problems back instead of an actual fix. My reply, verbatim: “Dude wtf.” It went back and fixed them.

AI is fast. Fast and correct are not the same thing.

The AI will grade its own homework, and lie about it. Look at the Replit incident. During a 12-day vibe coding experiment, SaaStr founder Jason Lemkin’s AI assistant ran destructive commands that wiped a database with records on 1,206 executives and over 1,196 companies.

Then it made it worse. It built a database of 4,000 fake user profiles and lied about a unit test passing, then told Lemkin the rollback was impossible. It wasn’t; the rollback actually worked. Lemkin said it was a demo app, but he still lost around 100 hours of work. That was a demo. Now imagine real customers.

The numbers say the same thing. A December 2025 CodeRabbit analysis of 470 GitHub pull requests found AI co-authored code averaged 10.83 issues per PR versus 6.45 for human-only PRs, about 1.7 times more overall, with up to 2.74 times more security vulnerabilities.

In May 2026, security firm RedAccess reported that out of roughly 380,000 publicly accessible AI-built assets, about 5,000 vibe-coded apps built for business use were leaking corporate data, many of them because privacy settings defaulted to public. Some were exposing medical records and a bank’s financial data.

Where real data lives, vibe coding doesn’t

At my day job, I work with school district data. Student records are protected by FERPA, a federal privacy law. You don’t get to vibe your way through code that touches that. You have to be careful about exactly what the code spits out, every field, every time.

Same with Tradion, the trading app I’m building. It connects to people’s real brokerage accounts and learns how each person trades. If I vibe coded it, I might never notice personal info leaking somewhere it shouldn’t. Or Tradion’s memory layer looking like it works while being wired up wrong, quietly giving bad answers instead of crashing. Those are the scariest bugs. The ones that don’t tell you they’re there.

And here’s one most people don’t see coming. When AI gets messy output from another model, its favorite fix is to slap a regex patch or a keyword filter on top. I’ve watched a regex meant to keep a model’s output in one shape silently cut off a real response the moment it didn’t match exactly, instead of failing loud.

Works in the demo. Breaks on the first weird edge case in production. A vibe coder accepts that patch without thinking twice. I banned it completely.

What I do instead: AI-assisted coding

I don’t blindly trust what AI gives me. Everything goes through gates, the same principle behind keeping an autonomous agent from doing something stupid: don’t beg the model to behave; build something that actually stops it.

It starts with an issue. Opus writes the plan. A separate agent, plan-adversary, reviews that plan against the actual codebase, and Sonnet builds it.

Then I have the work verified. Automated tests (functional, end-to-end, smoke tests, and Docker builds) run before anything gets pushed or merged, and only then does it ship. It’s the same shape as the eight-stage review pipeline I run on every change, just compressed.

This isn’t hypothetical, and none of it was vibe coded either. I keep a running account of every Claude Code session, and across 61 of them between July and September 2026, the AI’s own work would have reached production twice if nothing had checked it first.

A “fix” for cleaning up old storage would have let one user delete another user’s data; a separate review pass caught it, not the AI that wrote it. A database migration reached main bundled inside eight commits nobody had reviewed together, one production didn’t have yet; the build broke, and I applied it by hand.

A vibe coder skips every single one of those steps.

I also give the AI written rules. A few from Tradion:

Numbers come from data, not model memory. Prices, returns, dates all come from real sources. The math happens in code, then the model explains it. If data is missing, say so. Don’t fill the gap. I hold the same rule for every project, not just Tradion.
No control layers on top of the model. No regex hacks to “fix” output. No keyword blocklists. No homemade JSON repair.
Provider docs over memory. Check the official docs before touching any AI integration, because model names and features change all the time.

A vibe coder would never write rules like these. They don’t even know the AI makes up numbers.

One honest thing I have to call out. Georgia Tech researchers track security flaws caused by AI-written code. Their Vibe Security Radar is a live-updating tracker, and it attributed 227 of 289 confirmed CVEs to Claude Code as of its August 26, 2026 update. That’s the tool I use every day.

And that proves my point. The tool isn’t the problem. Skipping review is. Same tool, different process, totally different result.

Bottom line

Vibe coding is a sketch. Production is a building people live in.

Sketch all you want. Vibe code the prototype, test your idea, throw it away. But the moment real users or real data show up, the vibes are over.

So learn to code. Actually read what the AI gave you. Understand why it gave you that. Then decide. AI is a really fast junior developer, and you’re still the one who signs off.

Where do you draw the line between a prototype you’d vibe code and something you’d review line by line?

My Open Source AI Project: 347 Commits, No Users

Mon, 21 Sep 2026 00:00:00 GMT

On June 18, 2025, I ran my first commit: “RAGnetic Project Setup.” I spent the next four months building an open source AI project and never launched it.

347 commits, 22,000 lines of Python, about 100 API endpoints, and the only people who ever touched it were a couple of coworkers I asked to try it as a favor. That’s the whole outcome.

Here’s what building the wrong thing, for four months, taught me.

The project is called RAGnetic, and the code is still on GitHub if you want to see the receipts yourself.

The open source AI project I meant to build, and what I actually built

I worked in biotech for about two years, from 2023 to 2025. What I kept running into was process knowledge scattered across docs, wikis, and people’s heads, and no fast way to ask a question and get a real answer.

The idea was to consolidate that knowledge and embed it into whichever AI model you preferred, local or one of the frontier ones, and just talk to it whenever you needed information.

Point it at your documents, your databases, your APIs, describe an agent in YAML, and deploy it. “Complete ownership of your on-premise AI.”

That’s a specific problem. What I shipped was not a specific solution to it.

The README organizes RAGnetic into five pillars: Ingest, Adapt, Collaborate, Interact, Evaluate.

Ingest: PDFs, DOCX, CSV, web pages, code repos, live SQL, with PII redaction and hybrid BM25 + vector retrieval.
Adapt: hosted models (OpenAI, Anthropic, Gemini), local ones through Ollama, and LoRA fine-tuning wired directly into agent config.
Collaborate: multi-agent graphs on LangGraph.
Interact: a Docker sandbox for code, a SQL toolkit, web search, ArXiv, an HTTP toolkit.
Evaluate: benchmarking with retrieval metrics and an LLM judge.

Underneath all of that: FastAPI, Celery, Redis, Postgres, Alembic migrations, user and role management, API keys with rate limiting, and a credits and cost-accounting system with per-model pricing tables, the kind of thing you need once you’re actually paying attention to what an AI tool costs you.

Read that list again. Each bullet is somebody else’s entire company.

I built a generic platform for anyone who might one day want to talk to their documents, instead of the specific tool a biotech team needs to talk to their process docs. Somewhere between the idea and the code, the target quietly disappeared.

Lesson 1: I tried creating a platform instead of a product

Here’s the part that actually stings. I didn’t need to guess who my first users would be.

They were sitting next to me. A couple of coworkers agreed to try it, people who had the exact problem I’d started this to solve.

What I handed them was RAGnetic: five pillars, a YAML config language, a CLI, a dashboard, LoRA fine-tuning. Not “ask your process docs a question.” A platform, when what they needed was a product.

Here’s what I’ve come to think since: the best products don’t try to solve everything, at least not in the beginning. They solve one thing all the way, for one person, and earn the right to solve the next thing later. Mine tried to solve everything from day one, which meant it never quite solved anything.

Part of why it never got further than that was scope. One more feature was always the thing standing between where it was and where it needed to be to show someone.

Retrieval was fine, but the eval system wasn’t done. The eval system was fine, but the dashboard needed a rework.

There was always a next thing, and the next thing always felt more urgent than putting an unfinished tool in front of the people who’d asked for the specific thing in the first place.

The moment that actually landed the lesson wasn’t a user conversation. It was applying to YC and getting rejected.

That’s what made me look at the space honestly: RAG frameworks and agent platforms were already one of the most saturated categories in software, full of funded teams shipping the same pitch. A generic “better platform” story loses to whoever has more distribution.

I didn’t have distribution. I had a biotech problem and two coworkers who would have told me, for free, whether embedding their process docs into a model actually helped them find an answer faster. I built the platform instead of asking them.

What I’d do differently: skip the platform. Build the one thing the person next to me needed, show it to them in a week, and let their answer decide whether pillar two ever gets built.

Lesson 2: I built a benchmark that couldn’t fail

I had a sandboxed code executor, a distributed task queue, role-based access control, and a cost calculator with per-model pricing tables. What I didn’t have was a trustworthy answer to “is this agent any good?”

Benchmarking an AI agent means the same thing it means anywhere else: you write down a fixed set of test questions with known correct answers, run the agent against every one of them, and check two things whenever anything changes. Did it pull the right source material back out of your documents, and did the answer it gave actually match a correct answer?

Do that before the next feature, and you have a number that tells you whether a change made the system better or worse. Skip it, and every “it feels smarter now” is a guess.

Here’s how RAGnetic did it. The only benchmark I ever saved to the repo ran 10 questions, generated by an LLM from a single PDF (a paper on language agents). The gold answers were generated and then never compared to what the agent actually produced.

The benchmark only asks an LLM judge whether the answer is faithful to the retrieved context, and on that measure it scored 10 for 10. Worth knowing that’s a real metric and a real distinction, not something I made up to sound rigorous: Ragas lists faithfulness and factual correctness as separate metrics, because “grounded in the context you retrieved” and “actually the right answer” are different questions. I only ever asked the first one.

The retrieval side tells the real story. I logged two versions of the retrieval metrics: a strict one that matches chunk IDs exactly, and a lenient one with a content-hash fallback for when the IDs don’t line up.

The strict metric scored zero on every single question, all 10, because the two ID systems it was comparing never could have matched. The lenient metric hit on 9 of 10.

That means one question missed retrieval entirely, the agent answered from context that wasn’t the right context, and the faithfulness judge scored it faithful anyway.

What the benchmark measured	Result
Faithfulness: is the answer supported by the context it retrieved?	10 / 10
Retrieval, lenient (content-hash fallback match)	9 / 10
Retrieval, strict (exact chunk-ID match)	0 / 10
Whether the answer was actually correct	never measured

A 100% pass rate on 10 easy questions from one document isn’t evidence the system works. It’s evidence I never built a way for it to fail, the same trap an overfit model falls into: a number that looks perfect because you never tested it against anything that could prove it wrong.

I’ve made the same mistake in the other direction too: when I later looked at how Claude Code’s auto mode keeps an agent in check, the thing that stood out was a small, dumb judge sitting between the agent and the action, not a vibe check. A real evaluation is the same idea applied to quality instead of safety: something small and mechanical that can actually say no.

Meanwhile the README called the whole thing “production-ready” and “enterprise-grade,” under a section titled “Enterprise & Government Readiness.” One person, one PDF, one benchmark that couldn’t fail, and a claim like that sitting at the top of the file. It wasn’t dishonesty so much as the same instinct that built the five pillars: reach for the bigger claim before you have the evidence for the smaller one.

What I’d do differently: build the evaluation before the second feature. A real, hand-written test set on real documents, run on every change.

If gold answers exist, compare them in code, an exact match or a real scoring function, not just a second model’s opinion of the first model’s answer.

That’s a rule I hold myself to now on newer AI features: numbers and facts come from your actual data, checked in code, never from what a model remembers or another model’s vibe check. If you can’t measure “better” that way, you’re not steering, you’re guessing.

Lesson 3: every layer I sat on top of moved faster than I did

RAGnetic is glue over LangChain, LangGraph, Ollama, Hugging Face, PEFT and TRL, Chroma, FAISS, Qdrant, Pinecone, Mongo, Celery, Redis, and Postgres. Ten dependencies, all moving, all at once, and I was the only one keeping them wired together.

The scars are still in the repo. Eight requirements*.txt files, one of them literally named requirements-old-bloated.txt.backup.

A hard pin on numpy==1.26.4. A 2.4 GB virtual environment.

Commit messages like “More REDIS REDIS REDIS REDIS STUFF” and “SOME ANNOYING MIGRATION ISSUES THAT ARE FINALLY FIXED,” both from the same July week.

The clearest example is the Ollama client. app/core/config.py builds it by importing ChatOllama from langchain_community.chat_models.ollama. app/agents/agent_graph.py builds the same client by importing ChatOllama from langchain_ollama instead.

Two different code paths, constructing the same object, and one of them was already on the path LangChain had moved away from by the time I wrote the other — the current docs install langchain-ollama and import ChatOllama from it, not from langchain_community. I didn’t notice while I was building it. The ecosystem I was gluing together shipped a breaking change faster than I finished wiring the last one in.

Some of that is just what building on other people’s fast-moving projects looks like. But a lot of my time went into keeping the glue from coming apart, time I wasn’t spending on the one thing that would have made a coworker’s day easier.

What I’d do differently: fewer dependencies, one storage backend, and an install that takes one command. If a stranger, or a coworker, can’t try it in five minutes, they won’t.

Lesson 4: it kept me ahead of the AI race

I don’t want to end on pure regret, because that isn’t honest either.

The same speed that beat me in Lesson 3 cut the other way too. Chasing LangChain, LangGraph, Ollama, and half a dozen vector stores as they all moved at once meant I was never reading about the AI race from the sidelines.

I was inside it, learning and adapting in real time. Even as a failure, RAGnetic is the reason I know what the raw architecture of an agent actually looks like when you wire it together yourself instead of skimming a diagram of one.

In four months, alone, I shipped multi-tenant auth, background workers, a sandboxed executor, a hybrid retriever, a LoRA training pipeline wired into the same YAML as everything else, a CLI, a dashboard, a REST API, database migrations, and observability. I understand the full stack of an LLM application in a way that reading about it never would have given me.

I can also name the actual mistakes now instead of feeling vaguely bad about them: I built a platform before I found a person, I built a benchmark that couldn’t fail, and I let ten fast-moving dependencies eat time that should have gone to the two people who’d offered to try the thing.

RAGnetic is still up on GitHub, and I haven’t touched it since March. What I keep coming back to is whether the honest version of this idea is smaller than what I built: not a platform with five pillars, but something that runs entirely on your own machine, on an open-source model, built around the one question I started with. And what’s the proof?

If you’ve shipped something nobody used, what’s the one feature you kept promising yourself you’d add before showing anyone?

Claude Code Cost: 4 Rules That Work, 4 That Don't

Sat, 19 Sep 2026 00:00:00 GMT

Claude Code cost comes down to two things: how much context gets re-sent on every turn, and which model reads it. I checked the rules I use to keep both small against my own usage numbers and the docs as they read on September 19, 2026.

Four hold up. Four do not, and one of those, .claudeignore, is a file I can’t find anywhere in the docs.

One note on the word cost. I’m on the $200 Max plan, so I never see a bill.

What I spend is session and weekly limits, and these rules are about fitting more work under them. The dollar figures below are list-price estimates that I use as a measuring stick.

If you are on the $20 plan instead, how to get the most out of a $20 Claude plan applies the same ideas to Pro.

Here is the scorecard before the detail.

Rule	Verdict	Why
`opusplan`	Works	Sonnet types, Opus plans. Each switch re-reads the context uncached.
`/clear` with a `/wrap` skill	Works	Clearing costs nothing, and the notes keep continuity.
Haiku subagents	Works, with limits	The built-in Explore and Plan agents ignore the variable alone.
Disconnect unused MCP servers	Works, modestly	Their tool names still load: 2,659 tokens per request, measured.
`.claudeignore`	Not in the docs	The documented way is `Read` deny rules.
CLAUDE.md under 60 lines	Saves almost nothing	1.8k tokens, and it is cached.
`CLAUDE_CODE_MAX_OUTPUT_TOKENS`	No effect	A ceiling, not a bill.
`effortLevel: "high"`	Doesn’t save	Already the default on most models. On Opus 5.5 it may cost more than the `medium` default.

What actually drives Claude Code cost

I ran npx ccusage@latest over July to September 2026. It counted about 8 billion tokens, worth roughly $4,590 at list prices.

September alone, through the 19th, was 7.4 billion tokens and about $4,040. That is around twenty times the price of the plan.

ccusage prices at list rates and may not know the newest models, so read the dollars as estimates.

The shape of the numbers matters more than the total. Input outnumbers output 210 to 1. About 97 percent of that input is cache reads.

Using the standard multipliers, a cache read costs a tenth of a normal input token and a five-minute cache write costs 1.25 times.

My rough arithmetic says cache reads are about three quarters of my input cost and cache writes about a fifth. That is my math over ccusage output, not a published figure, and one-hour cache writes cost more than I assumed.

The docs explain why. In their words, Claude Code sends your full conversation with every request.

The cache makes the re-read cheap, not free. A session left open all day re-reads all of it for every one-line question.

So Claude Code cost is context size, times turns, times the price of the model reading it. Every rule below pushes on one of those three.

Three commands show where you stand. /usage shows the session’s prompt cache hit rate and, on a plan, which skills, subagents, plugins and MCP servers used your limits.

/context shows what is filling the window. ccusage shows the history.

My settings.json, minus the plugins

Here is the part of my settings.json that touches cost, with the plugins cut out.

{
  "model": "opusplan",
  "env": {
    "CLAUDE_CODE_SUBAGENT_MODEL": "haiku"
  },
  "hooks": {
    "SessionStart": [
      {
        "matcher": "clear",
        "hooks": [
          {
            "type": "command",
            "command": "cat \"$CLAUDE_PROJECT_DIR/SESSION_NOTES.md\" 2>/dev/null || true"
          }
        ]
      }
    ]
  }
}

Three things in there matter. model picks opusplan.

env sends subagents to Haiku. The hook prints my session notes after every /clear.

I left out four keys that are in my real file: CLAUDE_CODE_MAX_OUTPUT_TOKENS, CLAUDE_AUTOCOMPACT_PCT_OVERRIDE, effortLevel and advisorModel. They are the second half of this post.

Rule 1: opusplan, because Sonnet can type

"model": "opusplan" runs Opus in plan mode and Sonnet for execution, as the model docs describe.

I’m a software engineer, and I don’t vibe code. I know what I want changed before I ask, so Sonnet is more than enough to type it.

Opus earns its price on the planning step, when the question is what to build. The docs agree with the split: Sonnet “handles most coding tasks well and costs less than Opus.”

Here is my hot take. A lot of Opus spend is a very expensive way to work out what you want. If you already know, you don’t need the big model for most of the job.

There is one catch, and the caching docs are upfront about it. Each model has its own cache.

Under opusplan, every plan-mode toggle is a model switch, and the next request reads the whole conversation with no cache hits.

So opusplan is cheapest when the context is small at the switch. That is one more reason to start each task from a clear.

Rule 2: /clear is the cheapest command you have

The docs put it plainly: “When you want a fresh start instead of continuity, /clear costs nothing.” Stale context is re-sent with every message.

In one long session, /context showed 254k of the 290.8k tokens in use was the conversation itself, or 87 percent. The cheapest tokens are the ones you never carry into the next task.

The problem is continuity. Clear, and the session forgets what it was doing. I fixed that with a skill and a hook.

The skill, /wrap, writes a short notes file before I clear. The hook prints that file into the new session. Context dies, but continuity doesn’t.

The hook is the SessionStart block above. matcher: "clear" fires it only after a clear, and it prints SESSION_NOTES.md from the project root.

I have watched it fire: after a clear, the notes are right there at the top of the new session. I keep the file out of git with a global gitignore.

The /wrap skill that makes /clear safe

This is the whole skill, as it sits on my machine.

---
name: wrap
description: Save session notes before clearing
disable-model-invocation: true
allowed-tools:
  - Read
  - Write
  - Bash(git *)
  - Bash(cat *)
  - Bash(wc *)
---
Current notes:
!`cat SESSION_NOTES.md 2>/dev/null || echo "(none)"`

Git state:
!`git branch --show-current 2>/dev/null`
!`git log -1 --oneline 2>/dev/null`
!`git status --short 2>/dev/null | head -20`

Write SESSION_NOTES.md in the directory this session started in.
Replace it fully.
Do not create or edit any other file.

Rules:
- Max 40 lines, each under 120 characters. Run `wc -l` and trim if over.
- Merge with the current notes: keep decisions that still apply, drop
  finished items. Keep any other unfinished task under its own heading.
- Mark each done item VERIFIED (you saw the output) or UNVERIFIED.
- Never write secrets, API keys, tokens, connection strings, or .env values.
- Name every file, agent, and model. Never write "it". No em dashes.
- If nothing is left open, write only: "No open task. Last finished: <one line>"

Headings: Task, Branch and state, Done, Left to do, Decisions, Rejected (and why)

Reply with one line only: the line count and "Ready to /clear".

Four choices in there earn their place.

disable-model-invocation: true makes it manual. The skills docs say a skill set this way has no description in context at all, so /wrap costs nothing until I type it. /context agrees: wrap is not in my skills list, which runs to 9.9k tokens.

The ! lines run before Claude sees the skill. The command output replaces the placeholder, so the current notes and the git state arrive pre-loaded.

One gotcha from the docs: a failing command aborts the whole skill. That is why the notes line ends in || echo "(none)".

allowed-tools limits it to reading, writing, git, cat and wc. It has no reason to touch anything else.

The 40-line cap and “replace fully” are cost controls. The notes get printed into every fresh session, so a file that keeps growing is a tax I pay after every clear.

The cap forces a merge: keep what still applies, drop what is finished. Running the skill also costs little in cache terms, because the caching docs say a skill’s instructions are appended as a message and nothing earlier in the conversation changes.

Here is an example of what it writes. The project is made up.

## Task
Add rate limiting to the login endpoint.

## Branch and state
Branch feature/rate-limit, 3 commits ahead of main, working tree clean.

## Done
- VERIFIED: limiter added in auth.js, 5 attempts per minute per IP; 12 of 12 tests pass.
- UNVERIFIED: the Redis backend under load; only ran it locally.

## Left to do
- Load test at 200 requests per second.

## Decisions
- Fixed window, not sliding: simpler, and good enough for login.

## Rejected (and why)
- Per-account limits: they lock out real users during an attack.

Every done item is marked VERIFIED or UNVERIFIED, so the next session knows what was actually seen. The Rejected (and why) heading stops the new session from re-proposing ideas you already turned down.

Where to keep a skill

A skill is a folder with a SKILL.md file in it. The folder name is the slash command. Where you put the folder decides who gets it.

mkdir -p ~/.claude/skills/wrap     # personal: every project on this machine
mkdir -p .claude/skills/wrap       # project: commit it and your team gets it

Put the file in the folder and type /wrap. The description is recommended, and name defaults to the folder name.

wrap lives in my personal folder because I want it everywhere. A skill that only makes sense in one repo goes in the project folder instead.

Add that folder to .gitignore if you don’t want to share it.

Rule 3: Send subagents to Haiku, and know which ones ignore you

CLAUDE_CODE_SUBAGENT_MODEL: haiku sends subagent work to the cheapest model. Subagents matter for cost because verbose work, like test runs and log reading, stays in their context while only a summary comes back. The costs page recommends this.

There are two limits. The docs rank how a subagent picks its model: the per-invocation parameter, then the agent’s own model frontmatter, then this variable, then the main conversation’s model. And the subagent docs are blunt: setting the variable “by itself doesn’t change the model the built-in Explore and Plan subagents run on.”

That takes a second variable, CLAUDE_CODE_SUBAGENT_MODEL_FORCE=1, which I haven’t tried. I checked my own session. The built-in Explore agent ran on Opus 5.

Then there is the lesson. While I was checking these settings against the docs, I sent a subagent to look them up.

It ran on Haiku. It told me that CLAUDE_CODE_SUBAGENT_MODEL and CLAUDE_AUTOCOMPACT_PCT_OVERRIDE don’t exist. Both are in the docs.

A cheap model gave a confident wrong answer, and I only caught it because I re-checked the pages myself. Haiku is fine for reading files and running tests. I wouldn’t let it deliver the verdict.

Rule 4: Disconnect the MCP servers you aren’t using

I picked this one up recently: connected MCP servers cost tokens even when you never call them. That is true, so I measured it.

By default the cost is small. The MCP docs say tool search defers tool definitions, so “only tool names and server instructions load at session start.” The costs page tells you to run /mcp and disable servers you aren’t using.

Here is the measurement. I sent the same prompt in print mode on Haiku, once with my normal setup and once with --strict-mcp-config and an empty config. Then I added up the input, cache write and cache read tokens.

echo '{"mcpServers":{}}' > empty.json
claude -p "Reply with the single word: ok" --model haiku --output-format json
claude -p "Reply with the single word: ok" --model haiku --output-format json \
  --strict-mcp-config --mcp-config empty.json

My normal setup used 26,526 input tokens. With no MCP servers it used 23,867.

The difference is 2,659 tokens per request, identical on three runs, or about 11 percent of a bare session. That is the price of the names and instructions for everything I have connected, and it is re-read every turn at the cached rate.

Other models count tokens differently, so your number will differ.

/context gives the other half. In a long session on a 1M window it showed 290.8k tokens in use.

The MCP tools I had actually loaded took 4.1k. Another 163.1k of MCP tool definitions sat in a deferred pile that never entered the window, which is 16 percent of it.

That pile is what deferral keeps out, and ENABLE_TOOL_SEARCH=false would load it on every request.

Two more things stood out. First, the 23,867 is not MCP.

A bare “say ok” carries almost 24,000 tokens of Claude Code’s own prompt and whatever else I have installed. In an interactive session /context splits that as a system prompt of 4.8k, built-in tools of 15.5k, skills of 9.9k, custom agents of 1.5k and CLAUDE.md of 1.8k.

That is a different model and mode, so the totals won’t match.

Second, the small number depends on defaults. The docs say tool definitions load up front when ENABLE_TOOL_SEARCH is false, or auto while they total under 10 percent of the window.

They also load up front when a server sets alwaysLoad, or when ANTHROPIC_BASE_URL points to a non-first-party host.

I set none of those. If you do, the cost per server jumps.

Disconnecting is safe mid-session under the default. The caching docs say a deferred server connecting or disconnecting only appends to the conversation and leaves the cache alone. With deferral off, that change invalidates it.

What did not hold up

Four of my rules didn’t survive checking.

Hiding files takes a Read deny rule, not a .claudeignore

I thought I was hiding files with a .claudeignore. The permissions page never mentions one. What it describes is Read deny rules: add one for the path, “such as Read(./.env) or Read(./secrets/**).”

It gets worse. My user settings have no deny rules, this project’s deny list is empty, and there is no .claudeignore file in this repo either.

I believed I followed a rule that I didn’t. Here is what I’d add for this site, in .claude/settings.json.

{
  "permissions": {
    "deny": [
      "Read(./node_modules/**)",
      "Read(./dist/**)"
    ]
  }
}

Two caveats from the docs. Read deny rules cover the built-in file tools and file commands that Claude Code recognizes in Bash, like cat. They don’t cover grep -r pattern . or a script that opens files itself, and for Grep and Glob the docs call it a best-effort attempt.

And I haven’t measured how many tokens this saves. It keeps Claude out of files it has no business in.

That is safety first, savings maybe.

Deny rules are one wall. Claude Code auto mode adds another: a classifier that reviews risky actions before they run.

A 60-line CLAUDE.md saves almost nothing

My rule is that CLAUDE.md stays under 60 lines. Mine is 57 lines, and /context puts it at 1.8k tokens.

That is 0.2 percent of a 1M window, and about a fifth of what my skill descriptions take. It sits in the cached project context and gets read at the cached rate.

Against 8 billion tokens, the size is not where the savings are.

The docs’ own advice is to keep it under 200 lines and move workflow-specific instructions into skills, which load only when invoked. One more fact worth knowing: editing CLAUDE.md mid-session doesn’t invalidate the cache, but the edit doesn’t apply either. It loads on the next /clear, /compact or restart.

Keep it short because that is tidy. Just don’t count it as a saving.

Two settings that don’t save anything

CLAUDE_CODE_MAX_OUTPUT_TOKENS: 64000 is a ceiling. You pay for the tokens the model writes, not for the room it had.

I couldn’t confirm the variable in the docs, but it doesn’t matter. A cap you don’t hit costs nothing either way.

effortLevel: "high" is already the default on most models. The docs call it “the default on every model except Opus 5.5 and Opus 4.7.”

Opus 5.5 defaults to medium. The docs also say a top-level effortLevel in your user settings “doesn’t count for Opus 5.5,” but my own session log recorded high on its turns and /effort shows high, so check yours before you trust either. If the line does apply, it costs more on Opus 5.5, not less. On Opus 4.7, where the default is xhigh, it is a step down.

The level that saves is medium, which the docs say “reduces token usage for cost-sensitive work.” On most models each effort level also has its own cache, so pick one at the top of a session instead of switching halfway. Opus 5.5 is the exception: with a subscription or an API key, changing effort keeps the cache.

Autocompact at 200k: a trade I would not defend

I ran the auto-compact window at 200k for a while. The documented way is /autocompact 200k, which saves as the autoCompactWindow setting.

My real settings use CLAUDE_AUTOCOMPACT_PCT_OVERRIDE at 60, which is also documented. It sets the percentage of the window where compaction fires, and it can only compact earlier.

On models with a 1M window, the default is about 967K tokens. So 200k compacts almost five times sooner, and for me it sometimes compacted too early. That is the setting working as designed.

Does it cut Claude Code cost? On paper, partly.

Every turn re-reads the whole context, so a smaller window caps that. The caching docs also say that with a warm cache, a mid-session compaction “costs a fraction of what the context size suggests.”

Against that, compaction throws away detail and rebuilds part of the cache. If the cache has gone cold, the summary request reprocesses everything. I haven’t measured any of it, so I won’t claim a saving.

Here is what I’d do instead. Leave the window alone, clear between tasks, and run /compact by hand at a natural break. That matches the docs’ own tip: compact between tasks instead of waiting for auto-compaction to trigger mid-task.

What I would change first

Delete CLAUDE_CODE_MAX_OUTPUT_TOKENS. Set the effort level on purpose instead of restating the default. Swap the autocompact override for the default window.

Add the deny rules. Keep the four rules that work: opusplan, clearing with /wrap, Haiku subagents with their limits in mind, and a short MCP list. Run /mcp and disable anything you haven’t used this week.

advisorModel: "opus" is not a saving either. It picks the model that answers advisor calls, so it is a quality purchase. Keep it if it earns its place.

Then measure. Run /usage and read the Prompt cache (main) line, which shows your hit rate and misses. If a rule doesn’t move a number, it is a habit, not a saving.

The opusplan split above is also what runs my automated code review pipeline: a strong model for the review stages that matter, a faster one for everything else. That pipeline costs real minutes on top of everything in this post, and I think it’s worth it.

What is the one rule you follow to keep your Claude Code cost down, and have you checked it against the docs lately?

Linear Regression in Python: Study Hours vs Score With SciPy

Fri, 18 Sep 2026 00:00:00 GMT

Linear regression in Python takes one function call: scipy.stats.linregress(x, y). It fits a straight line through your data and hands back the slope, the intercept, and a few numbers that tell you how good the line is.

In this post I use it on made-up data: hours studied against exam score. Then I open the box. I’ll show what R-squared measures, how least squares finds the line, why a good line can still give a silly prediction, what changes when you add a second input, and how the same fit looks in scikit-learn.

Linear regression shows up in my Big Data Analytics master’s at SDSU: I used it in one of my final projects, before the AI boom, when you built and tested your own models. This post pulls that together with notes from a machine learning course on Udemy.

The example is mine: study hours against score. Every code block runs top to bottom on NumPy 2.2.6 and SciPy 1.15.3, and the output shown is what it printed. If arrays are new to you, start with my NumPy tutorial first.

What is linear regression?

Linear regression predicts a number, called the dependent variable y, from one or more inputs, called independent variables x. It assumes the relationship is a straight line:

y = b0 + b1 * x + e

b0 is the intercept: the predicted y when x is zero.
b1 is the slope: how much y changes when x goes up by one.
e is the error: everything about y that x doesn’t explain.

You already do this in your head. Say you earned $10,000 two years ago and $20,000 last year. If you guess $30,000 for this year, you just drew a line through two points and extended it.

The model answers two questions. Is there a linear relationship between x and y? And, with several inputs, which one matters most?

It has a big catch. It only works when the relationship really is roughly linear.

And it will never predict every point exactly. The error term is there for a reason.

How do you run linear regression in Python?

Install what you need:

pip install numpy scipy matplotlib

I generated 100 fake students so I’d know the true answer. Hours studied is normally distributed around 20. Each hour adds 1.1 points to the score, which is out of 100.

There’s also a second input, exercises completed, that adds 0.7 points per exercise, plus random noise.

For now I’ll ignore exercises and regress score on hours alone.

import numpy as np
from scipy import stats

rng = np.random.default_rng(1)

hours = rng.normal(20, 6, 100)
exercises = rng.normal(15, 4, 100)
score = 25 + 1.1 * hours + 0.7 * exercises + rng.normal(0, 4, 100)

print(f"hours: min {hours.min():.1f}, max {hours.max():.1f}")
print(f"score: min {score.min():.1f}, max {score.max():.1f}")

slope, intercept, r_value, p_value, std_err = stats.linregress(hours, score)
print(f"slope {slope:.3f}  intercept {intercept:.3f}  r {r_value:.3f}  r2 {r_value**2:.3f}  p {p_value:.2e}  stderr {std_err:.3f}")

hours: min 3.7, max 32.7
score: min 39.2, max 75.4
slope 1.077  intercept 35.103  r 0.770  r2 0.593  p 7.28e-21  stderr 0.090

Here’s what each number means. The SciPy docs list them all.

slope, 1.077: each extra hour of study goes with about one more point. That’s close to the 1.1 I built in. It says “goes with”, not “causes”.
intercept, 35.103: the predicted score at zero hours. Nobody in this data studied less than 3.7 hours, so don’t read too much into it.
r, 0.770: the correlation between hours and score.
p, 7.28e-21: the p-value for a test where the starting assumption is that the slope is zero. A tiny value means that if the true slope were zero, you would almost never see a slope this large by chance.
stderr, 0.090: the uncertainty on the slope.

What does R-squared tell you?

R-squared, also called the coefficient of determination, says how much of the variation in y your line captures. Here it’s 0.593, so the line explains about 59% of the spread in scores. The rest is everything else: exercises, sleep, a lucky guess.

SciPy’s docs confirm the shortcut: the square of rvalue is the coefficient of determination. But the formula is worth seeing once:

R-squared = 1 - (sum of squared errors) / (sum of squared distances from the mean)

The bottom part is the total variation in scores around their own average. The top part is what the line failed to explain. In code:

def predict(x):
    return slope * x + intercept

sse = np.sum((score - predict(hours)) ** 2)
sst = np.sum((score - score.mean()) ** 2)
print("1 - sse/sst =", round(1 - sse / sst, 3))

1 - sse/sst = 0.593

Same number. Two ways to read it:

0 means the line explains nothing. 1 means it explains everything. A low value is a poor fit. A high value is a good one.
It isn’t strictly stuck between 0 and 1. On new data, or for a model fitted without an intercept, R-squared can go negative. That means the model does worse than just guessing the average.

A high R-squared doesn’t prove your model is right. It just says the line follows the data you fitted it to.

How does least squares find the line?

The method is called ordinary least squares, or OLS. It picks the slope and intercept that make the total squared error as small as possible.

The error for one student is the actual score minus the predicted one. Then we square it, for two reasons:

A prediction can miss high or low. Squaring makes every miss positive, so misses can’t cancel each other out.
Squaring punishes big misses far more than small ones.

Squaring doesn’t guarantee a better model. It just defines what “best line” means.

You never have to do the math, because Python does it. But knowing what’s under the hood helps.

The answer has a closed form. For one input it looks like this:

slope     = r * (standard deviation of y) / (standard deviation of x)
intercept = mean of y - slope * mean of x

I checked it against SciPy:

slope_hand = r_value * score.std() / hours.std()
intercept_hand = score.mean() - slope_hand * hours.mean()
print("by hand:", round(slope_hand, 3), round(intercept_hand, 3))

by hand: 1.077 35.103

Identical. So OLS gets its answer in one step.

Some tutorials show the line inching toward the data over many iterations. That’s gradient descent, which is how neural networks are trained, and it isn’t needed for a straight line.

I use it in my overfitting vs underfitting post if you want to see it running.

How do you predict with the fitted line?

Multiply the input by the slope and add the intercept. I already wrote predict() above. For a student who studies 30 hours:

print("30 hours ->", round(predict(30), 1))
print("100 hours ->", round(predict(100), 1))

30 hours -> 67.4
100 hours -> 142.8

The first answer is reasonable. 30 hours sits inside the range of the data, which ran up to 32.7.

The second answer is nonsense. A score of 142.8 isn’t possible on a test out of 100.

The math worked perfectly. The line just has no idea that scores stop at 100, and it has never seen anyone study for 100 hours.

Only trust predictions inside the range you fitted.

To draw the data and the fitted line with matplotlib:

import matplotlib.pyplot as plt

plt.scatter(hours, score)
plt.plot(hours, predict(hours), c="r")
plt.xlabel("Hours studied")
plt.ylabel("Score")
plt.show()

What is multiple linear regression?

Multiple linear regression uses two or more inputs. Add exercises completed to the model and it becomes:

score = b0 + b1 * hours + b2 * exercises

Now there are three numbers to estimate, and linregress can’t do it, since it only takes one x. NumPy’s least squares solver does the same job for any number of inputs. Add a column of ones so the model gets an intercept:

X = np.column_stack([np.ones_like(hours), hours, exercises])
coef, *_ = np.linalg.lstsq(X, score, rcond=None)
b0, b1, b2 = coef
print(f"b0 {b0:.3f} b1 {b1:.3f} b2 {b2:.3f}")

pred = X @ coef
r2 = 1 - np.sum((score - pred) ** 2) / sst
print("multiple r2", round(r2, 3))
print("30h, 15 ex ->", round(b0 + b1 * 30 + b2 * 15, 1))

b0 24.737 b1 1.091 b2 0.686
multiple r2 0.739
30h, 15 ex -> 67.8

The estimates land close to the truth I built in: 25, 1.1, and 0.7. R-squared jumps from 0.593 to 0.739, because exercises really do matter in this data.

Each hour now adds about 1.09 points with exercises held fixed. In this data, hours and exercises are almost uncorrelated (-0.03), which is why the one-input slope of 1.077 was already close to the truth.

In real data, inputs usually overlap, and leaving one out shifts the slope of the other.

One warning. Adding an input can never lower R-squared on the data you fitted, even if that input is pure noise.

So a rising R-squared isn’t proof that a new variable belongs. Check on data the model hasn’t seen, which is the whole point of the overfitting vs underfitting comparison.

Where have I used linear regression for real?

In the SDSU program, my part of the BDA 594 final group project was about NFL special teams. We were given league data and asked to build new metrics for the specialists: punters, kickers, and long snappers. Then we ranked them, hoping to find something that could help a team perform better.

I preprocessed the exploratory data and built predictions for two things: the average distance a special teams player travels during a game, and their kick success rate in a game. The model was a simple linear regression, built as an artificial neural network.

That’s less odd than it sounds. A neural network with no hidden layers and no activation function is linear regression: one weight per input, one bias, and a weighted sum.

What changes is how it’s fitted. The network nudges its weights a little on every pass, which is gradient descent, instead of using the closed-form formula from earlier.

Both are chasing the same line.

Ranking real people from predicted values is where the warnings above stop being academic. R-squared tells you how much of the spread your line explains.

It doesn’t tell you whether a ranking is fair. And a prediction is only as good as the range the model was fitted on, so a player far outside that range gets a guess, not a prediction.

My grad projects are in this GitHub repo, with a short write-up of each one.

How do you do linear regression with scikit-learn?

Most tutorials on this topic use scikit-learn’s LinearRegression instead of SciPy. It gives the same answer. Install it with pip install scikit-learn, then reuse the hours, exercises, and score arrays from above.

from sklearn.linear_model import LinearRegression

model = LinearRegression()
model.fit(hours.reshape(-1, 1), score)
print(f"slope {model.coef_[0]:.3f}  intercept {model.intercept_:.3f}  r2 {model.score(hours.reshape(-1, 1), score):.3f}")
print("30 hours ->", round(model.predict([[30]])[0], 1))

slope 1.077  intercept 35.103  r2 0.593
30 hours -> 67.4

scikit-learn wants the inputs as a 2D array, with one row per student and one column per input. That’s why hours gets reshaped. Multiple regression is the same call with two columns:

X = np.column_stack([hours, exercises])
multi = LinearRegression().fit(X, score)
print(f"b0 {multi.intercept_:.3f}  b1 {multi.coef_[0]:.3f}  b2 {multi.coef_[1]:.3f}  r2 {multi.score(X, score):.3f}")

b0 24.737  b1 1.091  b2 0.686  r2 0.739

The slope, intercept, and R-squared match SciPy and NumPy to three decimals. So which one should you use?

SciPy’s linregress is the quickest for one input, and it also gives you the p-value and standard error.
scikit-learn is the better fit once you have several inputs, or when the model is one step in a bigger workflow. It has the same fit and predict pattern as the classifier in my KNN tutorial, so train and test splits and cross-validation work the same way.

The LinearRegression docs list the options.

What should you remember about linear regression in Python?

scipy.stats.linregress gives you the slope, intercept, correlation, and p-value in one call.
R-squared is the share of variation your line explains, and 1 - SSE/SST computes it.
OLS has a closed-form solution, so no iterations are needed.
Predict inside your data’s range. Outside it, the line is guessing.
For more than one input, use np.linalg.lstsq or scikit-learn’s LinearRegression.

What’s the first thing you’d try to predict with a straight line?

Overfitting vs Underfitting: How to Tell Which One You Have

Fri, 18 Sep 2026 00:00:00 GMT

Overfitting vs underfitting comes down to one comparison: error on the data the model trained on, against error on data it has never seen.

An overfit model scores great on its training data and badly on new data. It memorized the noise.

An underfit model scores badly on both. It was too simple to catch the trend in the first place.

Some of what’s below comes from my grad work in Big Data Analytics at SDSU, which was before the AI boom, when you built and tested your own models. The rest comes from notes I made in a machine learning course on Udemy.

The experiment is my own. Every number below is real output from NumPy 2.2.6, including the ugly ones.

What is overfitting vs underfitting?

Data has two parts: the pattern you care about and noise. Noise is the random, irrelevant variation that doesn’t mean anything.

Measurement error is noise. So is a student who had a bad morning.

Overfitting is fitting the noise. The model bends itself to hit every training point, including the ones that were just bad luck.

Training error comes out low. Error on new data comes out high.

In bias and variance terms, that’s low bias and high variance. Variance here means how much the fitted model would change if you resampled the training data.

Underfitting is the opposite. The model is too simple to follow the trend, so it misses even the training points.

Both errors come out high. That’s high bias and low variance.

A straight line through a curve is the classic case.

The trap is that training error alone can’t tell you which one you have. A model that memorizes every training point scores zero on the training data, and that number tells you nothing about how it will behave in the real world.

What does overfitting look like in Python?

Let’s fake some data so we know the right answer. The true pattern rises and then levels off.

I sample 15 training points, add noise with a standard deviation of 0.3, then draw 200 fresh points as a test set.

That noise sets a floor. Even a perfect model can’t beat the noise variance on average.

That’s the spread of the noise itself, 0.3 squared, or 0.09. So a test error near 0.09 is about as good as it gets, and a lucky test set can land a little under it.

I fit polynomials of different degrees to the same 15 points using NumPy’s Polynomial.fit.

Degree 1 is a straight line. Degree 14 has 15 coefficients for 15 points, so it can hit every one.

import numpy as np
from numpy.polynomial import Polynomial

rng = np.random.default_rng(7)

def true_curve(x):
    return 3 * (1 - np.exp(-5 * x))

x_train = np.linspace(0, 1, 15)
y_train = true_curve(x_train) + rng.normal(0, 0.3, 15)
x_test = rng.uniform(0, 1, 200)
y_test = true_curve(x_test) + rng.normal(0, 0.3, 200)

def mse(y, y_hat):
    return np.mean((y - y_hat) ** 2)

print("degree  train MSE  test MSE")
for degree in (1, 4, 14):
    model = Polynomial.fit(x_train, y_train, degree)
    print(f"{degree:>6}  {mse(y_train, model(x_train)):>9.3f}  {mse(y_test, model(x_test)):>8.3f}")

Here’s what it printed:

degree  train MSE  test MSE
     1      0.222     0.255
     4      0.029     0.086
    14      0.000     7.005

Read it row by row:

Degree 1 is underfitting. Both errors are high, and they’re close to each other. The line can’t follow the curve.
Degree 4 is the good fit. Test error is 0.086, right at the noise floor.
Degree 14 is overfitting. Training error is exactly zero, and test error is 7.005, about 80 times worse than degree 4.

I also swept every degree from 1 to 14. Training error only ever went down, from 0.222 to 0.000.

Test error fell until degree 4, crept up after that, and blew up past degree 11: 0.132 at degree 11, 0.626 at 12, and 7.005 at 14.

This is one random draw. I re-ran the setup with eight different seeds. The size of the blow-up changed a lot, but degree 14 always scored 0.000 on training and at least 2.0 on test.

How do you spot overfitting in a real model?

You can’t plot a model with fifty features and eyeball the curve. So you compare numbers instead. Split your data, then look at both errors together:

Both high, and close together: underfitting.
Training low, test much higher: overfitting.
Both low, and close together: a good fit.

That’s why supervised learning uses a train and test split, and often a third validation split. If you’ve followed my KNN classifier with scikit-learn, that’s the same reason the test set stays untouched until the end.

For neural networks and other models trained step by step, there’s a second view: error over time. One epoch is one full pass of the training data through the model.

Plot training error and validation error after each epoch and you get two curves. If neural networks are new to you, my TensorFlow tensors guide starts from the basics.

A healthy model shows both falling and levelling off. An overfitting model shows the training curve still falling while the validation curve turns upward.

When should you stop training?

At the point where validation error turns up. Everything after that is memorization.

To see it, I trained the same kind of model, a degree-14 polynomial, with gradient descent instead of solving for the best fit in one step.

The code writes it in a Chebyshev basis, which spans the same polynomials but keeps the numbers well behaved. Gradient descent nudges the weights a little after every epoch, which is how neural networks learn.

This time I used three sets. The training set fits the weights.

The validation set picks the stopping epoch. The test set only grades the final result, so it never influences a choice.

import numpy as np
from numpy.polynomial import chebyshev as C

rng = np.random.default_rng(7)

def true_curve(x):
    return 3 * (1 - np.exp(-5 * x))

x_train = np.linspace(0, 1, 15)
y_train = true_curve(x_train) + rng.normal(0, 0.3, 15)
x_val = rng.uniform(0, 1, 200)
y_val = true_curve(x_val) + rng.normal(0, 0.3, 200)
x_test = rng.uniform(0, 1, 200)
y_test = true_curve(x_test) + rng.normal(0, 0.3, 200)

DEGREE = 14
def features(x):
    return C.chebvander(2 * x - 1, DEGREE)

A_train, A_val, A_test = features(x_train), features(x_val), features(x_test)

def mse(y, y_hat):
    return np.mean((y - y_hat) ** 2)

w = np.zeros(DEGREE + 1)
learning_rate = 0.1
best_val, best_epoch, best_w = np.inf, 0, w.copy()

for epoch in range(1, 200_001):
    gradient = 2 * A_train.T @ (A_train @ w - y_train) / len(y_train)
    w -= learning_rate * gradient
    val_error = mse(y_val, A_val @ w)
    if val_error < best_val:
        best_val, best_epoch, best_w = val_error, epoch, w.copy()
    if epoch in (10, 1_000, 10_000, 50_000, 200_000):
        print(f"epoch {epoch:>7}  train {mse(y_train, A_train @ w):.3f}  validation {val_error:.3f}")

print(f"best validation error {best_val:.3f} at epoch {best_epoch}")
print(f"test error at that epoch {mse(y_test, A_test @ best_w):.3f}")
print(f"test error if we kept going {mse(y_test, A_test @ w):.3f}")

The output:

epoch      10  train 0.088  validation 0.442
epoch    1000  train 0.001  validation 0.317
epoch   10000  train 0.001  validation 0.138
epoch   50000  train 0.000  validation 0.368
epoch  200000  train 0.000  validation 2.646
best validation error 0.138 at epoch 10919
test error at that epoch 0.127
test error if we kept going 1.183

Three things stand out.

First, training error was already tiny by epoch 1,000 and stayed near zero. If I had only watched that column, I’d have said everything looked great.

Second, validation error kept improving long after training error stopped moving.

It bottomed out at epoch 10,919, then climbed to 2.646 by epoch 200,000 while training error sat at 0.000. The stopping point is where validation turns upward, not where training flattens.

Third, look at the test error. Stopped at the best epoch, the same model scores 0.127 on fresh data. Left running, it scores 1.183.

This trick is called early stopping. It’s a small fix with a big effect.

It also beat the one-step fit from earlier, which scored 7.005 on test. Same degree-14 family, very different result. Both scores come from 200 fresh points on the same curve.

How do you fix overfitting and underfitting?

The two need opposite fixes.

To fix underfitting, give the model more room:

Use a more flexible model, such as a higher polynomial degree or a deeper network.
Add features that carry real signal.
Train for longer, since an underfit model may simply not have converged yet.
Reduce any regularization you added.

To fix overfitting, take room away, or add evidence:

Get more training data. It’s the most reliable fix, and often the hardest.
Use a simpler model. Degree 4 beat degree 14 here.
Stop early, as above.
Add regularization, which penalizes large weights. For neural networks, dropout does something similar.
Use cross-validation, so one lucky split can’t fool you.

Most models have a dial for this. The number of neighbors k in KNN is one such dial.

With k = 1 each point is its own neighbor, so the model memorizes the training set. A large k smooths so much that it starts to underfit.

The simplest model of all is a straight line, and it’s a good place to start. I covered how it’s fitted in linear regression in Python. The scikit-learn docs also have a worked underfitting vs overfitting example if you want a second take.

Where does overfitting matter outside a toy problem?

Two of my SDSU projects sit right on this problem. Both are described in my grad projects repo.

The first was my individual final project for LING 583, a statistical text analysis class.

I used the 2020 US Election Tweets dataset from Kaggle to predict which candidate a tweet was about, from the text alone. I trained several different classifiers, then brought in the hashtags to see how they related to each candidate.

Comparing classifiers is a model-selection problem, and it’s the reason this whole post exists.

The only fair way to pick between them is how each one does on tweets it never trained on. A classifier with the best training score has told you nothing yet.

My notebook ended with a summary of the preprocessing, the models, the predictions, the limitations, and the results.

The limitations part is a habit worth keeping. A model that only reports its wins hasn’t really been tested.

The second was my part of the BDA 600 capstone. I built an LSTM for time series prediction on crypto data, and used the VADER sentiment analyzer on Twitter posts.

The project’s stated goal was a tool for cryptocurrency investors, and the team published the result as a StoryMap called Forecast and Predict Crypto Trend.

Time series overfit in an expensive way. A model can fit the past almost perfectly, and the past is exactly the data it has already seen.

The test that counts is forward in time. That’s why you split by date, not at random: train on the earlier dates and test on the later ones. A random split lets the model peek at the future.

What should you remember about overfitting vs underfitting?

Training error alone is deceptive. Always measure on data the model hasn’t seen.
Both errors high means underfit. Training low and test high means overfit.
More complexity never raises training error. It can raise test error.
Pick your stopping point, or your model, using a validation set. Grade once on the test set.
The noise floor is real. Chasing zero error means chasing noise.

Have you ever trained a model that looked perfect until it met new data, and what gave it away?

Claude Code Auto Mode: Lessons From the Source Leak

Tue, 31 Mar 2026 00:00:00 GMT

Claude Code auto mode does not beg the model to behave. It puts a small, fast judge between the agent deciding to act and the agent acting, and it only hands control back to you after repeated denials. I found that out the day the source leaked.

I woke up this morning to chaos on my timeline. Anthropic messed up. They accidentally shipped a 59 megabyte source map file to npm.

Just like that, 512,000 lines of unobfuscated TypeScript for Claude Code hit the open internet. Everyone immediately rushed to find the juicy stuff. Hidden features. Unreleased code names.

I did not care about that.

I went looking for one specific thing:

How do they stop the AI from doing something spectacularly stupid?

If you build AI agents, you know the fear. You write an auto loop script. You give an LLM your API keys. You go grab a coffee. You come back, and it has either deleted your database or hallucinated a massive cloud bill.

I ran into this exact nightmare while building my terminal platform, Tradion. I needed my trading agents to run autonomously, but I could not trust them with the big red button. It’s the same fear I didn’t solve well enough the first time, on the open source AI project I built and never launched.

Most developers try to fix this with begging. We write prompts like, “You are a careful assistant. Please do not wipe my hard drive.”

Begging an LLM does not scale.

When I cracked open the leaked source code, I found Anthropic knows this too. They do not beg. They built a three layer security funnel. And yes, they literally named part of it the “YOLO Classifier.”

Here is how it actually works, and what I learned.

The Problem With Claude Code Auto Mode

We all want the dream. We want to type “build a SaaS app,” hit enter, and go to sleep.

But you cannot just set a flag to dangerouslySkipPermissions: true and hope for the best.

Anthropic built a system inside yoloClassifier.ts. It sits between the AI deciding to do something and the AI actually doing it.

Here is the three step funnel they use.

Layer	Runs when	What it does
1: Fast path	Every action	Lets reads, in-project edits and known-safe actions straight through.
2: Fast judge	The action is risky, like a shell command	Sends a tiny payload to a constrained model that answers block or allow.
3: Slow judge	The fast judge flags it	A slower reasoning pass makes the final call. After a block, the agent can try a safer path.

Layer 1: The Fast Path

First, the system does a cheap check.

Is the action just reading data? Is it editing a normal file inside the project? Is it something already considered safe?

If yes, it goes through. No drama. No extra tokens. No slowdown.

Layer 2: The Fast Judge

If the action is risky, like running a shell command, it hits the YOLO classifier.

Anthropic does not send the entire chat history. That would burn money for no reason. Instead, they serialize the action into tiny JSONL style strings.

{"user":"fix the login bug"}
{"Bash":"rm -rf node_modules"}

That tiny payload gets sent to a second, heavily constrained model. This model has a strict token budget. It is not there to brainstorm. It is not there to be creative. It is there to look at the action and make a binary judgment.

Block or allow.

It acts like a mechanical circuit breaker.

Layer 3: The Slow Judge

If the fast model flags the action, the system escalates.

Now the same payload goes to a slower reasoning pass that takes a more careful look at why the command might be dangerous before making the final call.

And this is the important part: if the action gets blocked, the system does not always instantly stop and hand control back to the user. It can deny the action and let the agent try a safer path first. Only after repeated denials does it escalate back to the human.

That part is smart.

Anthropic describes the same two-stage design in its engineering write-up on auto mode. A fast single-token filter runs first, and the careful reasoning runs only if that filter flags the action. The same write-up puts the escalation point at three denials in a row or 20 in total. The permission modes docs say the same about auto mode pausing and prompting you again.

My Hot Take

You cannot rely only on prompts for safety.

If you want an AI to act autonomously, you need actual walls.

You need a primary agent that does the creative work. But right next to it, you need a tiny, fast, brutally simple judge.

The judge does not write code. It does not talk to the user. It does not explain itself for six paragraphs.

It looks at an intended action and says yes or no.

That is it.

What You Should Do Next

Stop trusting your primary agent.

If you are building an AI app, a stock bot, or a coding assistant, build your own version of this. Route dangerous actions through a fast, cheap model. Make it judge the action, not the vibes.

Safety is not about nerfing the AI. It is about building gates, escape hatches, and escalation paths.

That is the real lesson here.

Running Claude Code all day also has a price tag, which I break down in Claude Code cost: 4 rules that work, 4 that don’t. The same “don’t beg the model, build a wall” reasoning is what my automated code review pipeline is built on, just aimed at merges instead of shell commands.

How are you keeping your own agents from doing something spectacularly stupid?

TensorFlow in Python: How to Create Tensors

Wed, 27 Nov 2024 00:00:00 GMT

TensorFlow is hands down one of my all-time favorite deep learning tools! I was going to call it an AI tool, but let’s be real, ’AI tools’ these days make people think of image generators, essay writers, or tools to generate one of those weird videos of Will Smith eating spaghetti. Before the ‘AI Boom’ days of 2018–2019, calling something an AI tool meant it was a coding framework to build AI applications. Funny how the meaning has shifted over the years, huh? But that is not what I am writing about today. Today we will learn how to create tensors using TensorFlow in Python.

This TensorFlow in Python guide is primarily for people who are new to deep learning or machine learning frameworks, but I still expect you to have a decent knowledge of Python programming.

And if you are a complete beginner to the world of Machine Learning, then here are some of the articles that you may like:

KNN Algorithm Using Scikit-Learn – Classifying Iris Species (Tutorial)

We have a lot of material to cover today, so let’s get started.

What is Deep Learning?

Before we understand deep learning. Let’s understand what is machine learning.

Machine learning is the process of writing code that allows machines or computers to learn from observational data and make predictions based on that.

The best part is we don’t have to hand-feed machines every single instruction to make predictions. Past data or observational data is the main ingredient we need. The computer figures out the patterns on its own and gets smarter with experience.

Now, here’s where things get interesting. Deep learning is a subset of machine learning, but it cranks things up a notch.

While traditional machine learning works with structured data and requires you to select the important features manually, deep learning takes a more hands-off approach.

It uses artificial neural networks, a fancy term for algorithms inspired by how our brains work to automatically find patterns and features through data. ANN or artificial neural networks are capable of learning from data that is unstructured or unlabeled.

Why use Deep Learning?

Deep learning isn’t just a buzzword. It’s the engine behind the current AI boom that we are going through. It’s driving innovations from large language models that feel eerily human to self-driving cars that actually stay in their lane.

But why is deep learning stealing the show?

The Traditional Approach: Long Lists of Rules

Once upon a time, solving problems with computers meant creating endless lists of rules. Want to identify whether an image contains a cat? You’d have to manually code rules like:

“If it has pointy ears and whiskers, it’s probably a cat.”
“If it meows, chances are high.”
“If it scratches your sofa, definitely a cat.”

Sounds exhausting, right? And that’s just one problem. What happens when you want to expand your rules to dogs, birds, or, heaven forbid, giraffes? The traditional approach collapses under its own complexity.

Deep learning eliminates this hassle. Instead of you crafting every rule, the model learns the rules by itself from data.

Deep Learning Adapts to New Challenges

The world isn’t static, and neither are the problems we need to solve. That’s where deep learning truly shines. Unlike traditional methods, which struggle with changing environments, deep learning models can be retrained or updated as new data rolls in.

Let’s say you’ve trained a model to recognize fruit. Now, someone asks you to identify not just apples and oranges, but 101 different kinds of food. Creating rules manually for all that? No thanks. With deep learning, you feed the model new data, and it does the job for us.

Discovering Insights in Massive Data Collections

Here’s the thing, we’re drowning in data. Text, images, videos, you name it. Hidden in those terabytes are valuable insights waiting to be uncovered. But imagine trying to manually sift through all that information to find patterns. You’d go gray before you get anywhere.

Deep learning, on the other hand, thrives on this kind of challenge. Whether it’s detecting diseases in medical images or identifying customer trends, deep learning can handle the heavy lifting. It’s like having a detective who never sleeps and gets smarter with every case.

Real-World Applications That Prove the Point

If you’re still wondering why deep learning is worth the hype, just look around:

Voice Assistants: Siri and Alexa don’t magically understand your accent, they rely on deep learning.
Self-Driving Cars: Deep learning helps them recognize stop signs, avoid pedestrians, and navigate changing traffic conditions.
Personalized Recommendations: Ever wondered how Netflix knows what you want to watch next? Yep, deep learning.

These systems wouldn’t work nearly as well, or maybe at all with traditional methods.

Whether you’re dealing with complex problems, mountains of data, or rapidly changing environments, deep learning offers solutions that traditional approaches simply can’t match.

When not to Use Deep learning?

Deep learning is the superstar of the AI world, but it’s not always the right choice. Believe it or not, there are times when you should put the deep learning hype aside and go for a simpler solution.

When You Need Explainability

Deep learning models learn patterns that are often uninterpretable by humans. For instance, if you’re in a field like healthcare or finance, where decisions must be transparent and explainable, deep learning can be a dealbreaker.

Like seriously can you imagine telling a patient, “The AI says you need this treatment, but I can’t tell you why.”. Not something I would want to hear as a patient.

When a Traditional Approach Works Just Fine

If you are a fan of a show known as The Office, you may heard Michael Scott saying to Dwight, keep it simple stupid or aka K.I.S.S. And this mantra goes well for deep learning. You don’t have to overcomplicate things. If you can solve your problems with a basic rule-based system, why bother with deep learning?

For example, if you’re sorting emails based on whether they contain the word “urgent,” a few lines of rule-based code will do the trick. Seriously, no need to unleash a neural network on such a straightforward task.

When Errors Are Not an Option

Although deep learning models are powerful, they aren’t perfect.

Their outputs can be unpredictable, and mistakes are bound to happen. That’s why its important to use a more predictable approach for tasks that are mission-critical.

When You Don’t Have Much Data

This is probably my favorite one. No data? then no AI.

AI, machine learning, or deep learning heavily depends on data. Especially deep learning. It is hungry for data, like a lot of data. So, without enough data or training examples, these models tend to underperform, overfit, or just simply fail.

If you’re working with a small dataset, you’re better off sticking to traditional machine learning or even statistical methods. Deep learning’s real strength comes into play with large datasets where it can learn complex patterns.

When to Ask, “Is Deep Learning Worth It?”

Before diving headfirst into deep learning, ask yourself these questions:
Do I need explainable results? If yes, look elsewhere.
Is my problem complex enough to justify it? If not, save yourself the trouble.
Can I afford some errors? If no, steer clear.
Do I have enough data? If not, don’t even think about it.

Knowing when to use fancy tools like deep learning is great, but knowing when not to use them is even better. Sometimes, keeping things simple isn’t only just smart, but it can save you from disastrous outcomes.

What are Neural Networks?

We can’t really start coding neural networks unless we know what they are. I won’t be diving into the specifics of neural networks in this guide.

But for this tutorial, let’s at least understand the basic concepts and workflow that you need to know.

By definition, a neural network is a fancy term for algorithms mimicking the biological neurons of a human brain. These algorithms are the heart of any deep learning model and their purpose is to figure out complex patterns.

But how exactly do they work?

Turning Data Into Numbers

Before you can feed the data into a neural network, we need to translate the data into numerical values that the model understands.

For example, converting a picture of a flower into numerical pixel values. Or transforming a sentence into embeddings or tokenized numbers. And even sound waves can become a series of numerical amplitudes.

These numbers, representing your data, are fed into the neural network, which learns patterns, features, and relationships to solve the problem at hand. Whether it’s recognizing objects in images, understanding text, or converting audio to text, the process starts with data becoming numbers.

How Does Neural Network Work?

To keep things simple, I want you to think of it as a three-step process. We can talk about the specifics of these steps later.

But here is the breakdown:

Feed the Data: Once the data is turned into numbers, it’s fed into the neural network.
Learn Representations: The neural network identifies patterns or features in these numbers, like colors, edges, or shapes in an image.
Produce Outputs: It transforms the patterns into representation outputs. These might not make sense to us yet. But they’re also just numbers representing what the network has learned.

Turning Outputs Into Human-Readable Information

The neural network’s output is a numerical representation of the patterns it found. It’s up to us to convert this into something meaningful.

For example, you feed in a picture of a flower. The neural network processes it, discovers patterns like petal shapes and colors, and produces a representation output (e.g., “It’s 85% likely to be a rose”). Then you can interpret that output and label it as rose.

Types of Learning in Neural Networks

Neural networks can learn in different ways depending on how much information you give them about the data.

Supervised Learning

What it is: The neural network learns from labeled data or data already paired with the correct answer or label.

Example: You have images of flowers labeled with their names, like “rose” or “tulip.” The network learns to predict labels based on the patterns in the images.

Semi-Supervised Learning

What it is: The network has a mix of labeled and unlabeled data.

Example: You have 10,000 flower images, but only 1,000 are labeled. The network learns from the labeled images and uses that knowledge to make predictions on the unlabeled ones.

3. Unsupervised Learning

What it is: The network only has data and no labels. It finds patterns or structures on its own.

Example: You give the network a dataset of flower images. It might group them into clusters based on similarities (e.g., grouping roses together) without knowing their names.

4. Transfer Learning

What it is: The network takes what it learned from one task and applies it to a new, similar task.

Example: Training a neural network to recognize animals in photos can be adapted to identify flowers with much less training data.

Why Are Neural Networks So Powerful?

Neural networks shine because they can automatically learn patterns and relationships in data. Especially data that are complex, unstructured, or huge in size. They’re adaptable to new problems, can generalize across tasks, and make life easier by handling data that would be technically impossible for humans to analyze manually.

Anyway, let’s talk about Tensorflow. A powerful deep learning library that I just can’t get enough of.

TensorFlow in Python: The Toolkit for Deep Learning Greatness

Now that we’ve talked about neural networks and how they work their magic, let’s dive into the real MVP of deep learning, which is TensorFlow.

I have been using TensorFlow in Python since 2018 and it has been a regular tool in my arsenal for deep learning because it combines simplicity with power. This library is a game changer for building, training, and deploying deep learning models. If you are not using TensorFlow yet, you are missing out on one of the best software libraries in the AI world.

What Is TensorFlow?

TensorFlow is a complete machine learning platform that takes raw data and transforms it into a fully deployed deep learning model. It supports coding in Python and other languages while enabling you to run your models on specialized hardware like GPUs and TPUs.

The library also includes TensorFlow Hub, a collection of pre-built models that you can easily integrate into your projects.

Whether you are solving tasks in computer vision or natural language processing, TensorFlow provides a head start by offering models we can utilize for specific use cases.

Why TensorFlow?

TensorFlow comes with tons of features that make it stand out from the crowd. Here’s why it’s a favorite for both beginners and experts:

1. Easy Model Building

TensorFlow simplifies the process of creating complex neural network architectures. Its Keras API allows you to design and train models with minimal effort while maintaining flexibility for customization.

2. Scalable Production

It supports deploying models across multiple platforms, including mobile devices, web applications, and even edge devices. For example, training a computer vision model to recognize objects in a smartphone app to identify items in real-time.

3. Powerful Experimentation

Researchers use TensorFlow in Python for its flexibility and capability to test experimental architectures. Moreover, TensorFlow provides robust tools for iterating on models and fine-tuning them for specific tasks.

4. Cost Efficiency

TensorFlow optimizes training by leveraging GPUs and TPUs, significantly reducing the time and computational cost required for deep learning tasks. Additionally, being open source makes TensorFlow in Python an accessible tool for anyone, regardless of budget.

Hardware Support: GPUs and TPUs

TensorFlow utilizes special hardware that accelerates deep learning workflows. Here is a quick overview of GPUs and TPUs:

The primary use of GPUs were originally for rendering graphics, but they’re insanely good at crunching numbers. When it comes to deep learning, GPUs can perform thousands of numerical calculations simultaneously, making them perfect for training neural networks. And, TPUs are even faster than GPUs when handling TensorFlow tasks and are often used for large-scale machine learning projects. Google created TPUs specifically for TensorFlow. These are advanced chips optimized for AI workloads.

Tensors in Neural Networks

In the neural network section we discussed about inputs such as images or text and how we turned those inputs into numerical encodings, fed them into the neural network, and then got outputs that we could interpret.

Well, here’s the thing, both those numerical encodings (inputs) and the representational outputs (outputs) are actually tensors. Sounds easy right?

A Tensor is simply a numerical representation of data. It’s that simple. It is how we encode information into a form that a neural network can process. Whether that information is an image, text, audio or any other type of data. I like to think of tensors as the universal language of deep learning.

How Tensors Work in Neural Networks

Here’s a little break down on how the whole process works:

1. Input as a Tensor: When we encode raw input data (like an image of a flower) into numbers, we are essentially turning it into a tensor.

2. Processing the Tensor: The tensor passes through the neural network. The network analyzes the numbers, learns patterns, and figures out what matters most in those numbers.

Output as a Tensor: The network outputs another tensor, which represents the patterns it has learned from the original input tensor.

4. Interpreting the Output: Finally, we take the output tensor and convert it into something that we humans can understand. For instance, like a label (“this is a rose”).

Tensors are how deep learning models see the world. They are the numerical form of data, allowing neural networks to analyze and learn from inputs. Whether it is finding patterns in an image or interpreting the meaning of a sentence, tensors make it possible.

Creating Tensors with TensorFlow in Python

Now that we understand what tensors are, let’s jump into creating them using TensorFlow. For this tutorial I will be using Google Colab.

So open your browser and head over to https://colab.research.google.com/

Colab is a free cloud-based Jupyter Notebook that lets you run Python code and store your projects on Google Drive.

The best part? TensorFlow already comes with it, and you can even use free GPUs. It is perfect if you want to skip the hassle of setting up TensorFlow on your local machine.

If a window pops up, go ahead and click on New Notebook on the bottom left corner.

After that name your notebook and click the Connect button at the top right to start coding.

Creating Tensors with tf.constant()

First, we need to import TensorFlow in Python. Run the following:

import tensorflow as tf

We are using the alias tf, which is the standard convention for TensorFlow in Python, but you can choose any alias you like.

Now that TensorFlow is ready to go, let’s create our first tensor.

Creating a Scalar Tensor

Let’s start with the simplest tensor, scalar:

scalar = tf.constant(4)
scalar

When you run the code, you get:

<tf.Tensor: shape=(), dtype=int32, numpy=4>

Here’s what the output tells us:

tf.Tensor: It’s a tensor object.
shape=(): It’s a scalar, so there’s no shape (empty).
dtype=int32: The data type is 32-bit integers.
numpy=4: The actual value of the tensor.

Now I want to check the number of dimensions (or ndim) of our scaler:

scalar.ndim

Output: 0

A scalar has no dimensions, which is why the ndim is 0.

Creating a Vector Tensor

Now let’s move to a vector, which has one dimension.

Here we will pass a Python list into tf.constant() to create it:

vector = tf.constant([8, 8])
vector

Output:

<tf.Tensor: shape=(2,), dtype=int32, numpy=array([8, 8], dtype=int32)>

Here’s what’s different from the scalar we saw earlier:

shape=(2,): The vector has two elements.
numpy=array([8, 8]): A list of two numbers.

Now let’ts check the dimensions:

vector.ndim

A vector has one dimension, the shape (2,) indicates.

Creating a Matrix Tensor

A matrix has two dimensions. Let’s create one:

matrix = tf.constant([[4, 5], [5, 4]])
matrix

Output:

<tf.Tensor: shape=(2, 2), dtype=int32, numpy=
array([[4, 5],
       [5, 4]], dtype=int32)>

The shape (2, 2) means we have two rows and two columns.

To check the dimensions:

matrix.ndim

Output:

Our matrix here has two dimensions.

Specifying Data Types

By default, tensors created with tf.constant() have the dtype=int32.

Let’s specify a different data type:

newer_matrix = tf.constant([[4., 5.], [5., 4.], [6., 7.]], dtype=tf.float16)
newer_matrix

Output:

<tf.Tensor: shape=(3, 2), dtype=float16, numpy=
array([[4., 5.],
       [5., 4.],
       [6., 7.]], dtype=float16)>

The dtype=float16 means the tensor uses 16-bit precision, which takes up less memory than int32.

If you ever encounter a data type error in the future, you can adjust it using the dtype parameter.

Creating a 3D Tensor

Let’s level up and create a 3-dimensional tensor:

three_dimension_arr = tf.constant([[[1, 2, 3],
                                    [2, 3, 4],
                                    [4, 5, 6]]])
three_dimension_arr

Output:

<tf.Tensor: shape=(1, 3, 3), dtype=int32, numpy=
array([[[1, 2, 3],
        [2, 3, 4],
        [4, 5, 6]]], dtype=int32)>

The shape (1, 3, 3) indicates one array containing three rows and three columns.

Checking the dimensions:

A three-dimensional tensor has three dimensions.

Relationship between Shapes and Dimensions

Let’s summarize the relationship between shapes and dimensions:

Scalar: A single number with no dimensions (shape=() and ndim=0).
Vector: A one-dimensional array (shape=(n,) and ndim=1).
Matrix: A two-dimensional array (shape=(n, m) and ndim=2).
Tensor: An n-dimensional array, where n can be 0, 1, 2, or higher.

When I first started learning about tensors, thinking of tensors as levels also helped me. This is an approach that I followed:

0D Tensor: Scalar (a single number).
1D Tensor: Vector (a list of numbers, like car speed and direction).
2D Tensor: Matrix (rows and columns).
3D Tensor: 3D Array (layers of matrices).

Now that you know how to to create tensors with tf.constant(), we will explore another way of creating tensors, tf.Variable()

Creating Tensors with tf.Variable()

So far we’ve covered tf.constant(). These are unchangeable tensors, let’s look at its sibling, tf.Variable().

This one’s a bit different because it allows you to create changeable tensors. These types of tensors are perfect for when you need flexibility in your deep learning models.

What Exactly Is tf.Variable()?

A tf.Variable() tensor is mutable, meaning you can change its values after it’s creation. This sets it apart from tf.constant(), which creates unchangeable tensors.

tensor1 = tf.Variable([4, 5])
tensor2 = tf.constant([4, 5])

print("Tensor 1: ", tensor1)
print("Tensor 2: ", tensor2)

Output:

Tensor 1:  <tf.Variable 'Variable:0' shape=(2,) dtype=int32, numpy=array([4, 5], dtype=int32)>
Tensor 2:  tf.Tensor([4 5], shape=(2,), dtype=int32)

At first glance, both look pretty similar. But notice how Tensor 1 uses tf.Variable, while Tensor 2 uses tf.constant. The difference becomes clear when we try to modify their values.

Modifying a tf.Variable() Tensor

Let’s start by indexing Tensor 1:

tensor1[0]

Output:

<tf.Tensor: shape=(), dtype=int32, numpy=4>

This gives us the value 4 at index position 0.

Now, let’s try to change the value at index 0:

tensor1[0] = 8
tensor1

Output:

---------------------------------------------------------------------------
TypeError: 'ResourceVariable' object does not support item assignment

Oops! That didn’t work. Directly assigning a value doesn’t work for tf.Variable objects. But don’t worry, there’s another way.

Using .assign() to Modify Tensors

To change the value of a tf.Variable tensor, you need to use the .assign() method. Here’s how:

tensor1[0].assign(8)
tensor1

Output:

<tf.Variable 'Variable:0' shape=(2,) dtype=int32, numpy=array([8, 5], dtype=int32)>

Success! We replaced the value 4 with 8 using .assign().

Using .assign() on tf.constant()

Let’s try modifying Tensor 2, created with tf.constant(), using .assign() to see what happens:

tensor2[0].assign(8)
tensor2

Output:

---------------------------------------------------------------------------
AttributeError: 'tensorflow.python.framework.ops.EagerTensor' object has no attribute 'assign'

Thus, we get an attribute error. That’s because tf.constant() creates immutable tensors, so their values can’t be changed after creation.

When to Use tf.Variable() vs tf.constant()

Use tf.Variable() when your tensor needs to change during computation, such as in training models where weights and biases are updated.
Use tf.constant() for fixed data, like configuration values or inputs that don’t need to change.

Conclusion

If you are interested in working with deep learning, then TensorFlow in Python isn’t just another library you’ll occasionally use. It is your trusted resource for building, experimenting, and deploying AI models. And the best part is, Google made it absolutely free and open source for us to use.

In short, tensors are the backbone of deep learning, and understanding them is the first step to unlocking TensorFlow’s full potential. So go ahead, try out the examples, play with the dimensions, and experiment with what you’ve learned. The more you practice, the closer you’ll get to TensorFlow wizardry (okay, maybe not wizardry, but close enough).

TensorFlow is everchanging and evergrowing. Despite using it for the last 6 years, I am constantly learning new things about TensorFlow in Python. What we just learned here is probably the tip of an iceberg. There is so much to learn about TensorFlow and I plan to write about them on my blog as much as i can. So stay tuned and make sure to check out my other guides.

150 Python Interview Questions You Should Know

Sun, 24 Nov 2024 00:00:00 GMT

So you have a Python interview coming up or you are interested in positions that require you to be a hotshot at Python? Maybe you are just a curious bird like me and looking for python interview questions to sharpen your coding skills. Whatever it is, you are at the right place. Because in this post I will list not just 10, 20 or 30 but 150 Python interview questions and their answers that are packed to help you ace your next Python interview.

Python Interview Questions and Answers

1. What is Python?

Python is a high-level, interpreted, general-purpose programming language known for its simplicity and readability.

2. Who created Python, and what language did it succeed?

Python was created by Guido van Rossum in 1991. It is the successor of the ABC programming language, which inspired its design.

3. Is Python compiled or interpreted?

Python is an interpreted language, meaning it executes code line by line, making it easy to debug.

4. What are some key features of Python?

User-friendly syntax.
Object-oriented.
Dynamically typed.
Portable and platform independent.
Extensive standard library.

5. How is Python’s syntax different from Java’s?

Python uses indentation to define code blocks, whereas Java relies on curly braces {}. This makes Python more readable and concise.

6. What’s the difference between a list and a tuple in Python?

Lists are mutable (modifiable), while tuples are immutable (non-modifiable). Lists use square brackets [] and tuples use parentheses (). Tuples are generally faster and more memory-efficient than lists.

7. Explain local and global variables in Python.

Local variables: Declared inside a function and accessible only within that function.
Global variables: Declared outside any function and accessible throughout the program.

8. What is type conversion? Name some methods used in Python.

Type conversion refers to converting one data type to another. Common methods include:

int()
float()
str()
tuple()
list()
set()

9. How do you create a function in Python?

Functions in Python are created using the def keyword:

def greet():
    print("Hello, World!")

10. What is a default argument?

A default argument is a predefined value assigned to a function parameter. If no value is passed during the function call, the default value is used.

11. What are anonymous functions in Python?

Anonymous functions, also known as lambda functions, are single-expression functions defined using the lambda keyword.

add = lambda x, y: x + y
print(add(5, 3))

Output: 8

12. Name two other scopes in Python apart from local and global.

Module-level scope: Variables accessible throughout the module.
Built-in scope: Contains Python’s built-in functions like print() and len().

13. What are Python’s built-in collections?

List
Tuple
Set
Dictionary

14. How do you comment on a single line in Python?

Use the # symbol:

# This is a single-line comment

15. How do you comment on multiple lines in Python?

Use triple quotes ‘’’ ‘’’:

'''
This is a multi-line comment.
It spans multiple lines.
'''

16. Write a loop to iterate over a list in Python.

my_list = [1, 2, 3, 4]
for item in my_list:
    print(item)

17. What’s the difference between break and continue?

break: Terminates the loop entirely.
continue: Skips the rest of the current iteration and moves to the next.

18. What is the pass statement in Python?

The pass statement acts as a placeholder and does nothing when executed. Unlike comments, it’s required for code structure in some places.

19. What is a dictionary in Python?

my_dict = {"name": "Alice", "age": 25}

20. Which Python collection does not allow duplicate values?

Sets do not allow duplicate values.

21. How do you check if a string starts with a capital letter in Python?

Use the istitle() method:

print("Hello World".istitle())

Output: True

22. How do you check if a substring exists in a string?

Use the in operator:

text = "Welcome to Python programming."
print("Python" in text)

Output: True

23. What built-in function is used to check the length of a string?

The len() function is used to determine the length of a string.

text = "Python"
print(len(text))

Output: 6

24. How do you convert a string into a list?

Use the split() method to break a string into a list of substrings. For more on working with strings, see Python text editing and processing.

text = "Python is amazing"
words = text.split()
print(words)

Output: [‘Python’, ‘is’, ‘amazing’]

25. Write a Python code snippet to reverse a string.

text = "Python"
reversed_text = text[::-1]
print(reversed_text)

Output: nohtyP

26. What happens when you add a string and an integer in Python?

It raises a TypeError because Python does not allow implicit type conversion between strings and integers.

result = "Python" + 123

Output: TypeError

27. How do you check if all characters in a string are uppercase?

Use the isupper() method.

text = "PYTHON"
print(text.isupper())

28. What is the difference between range() and xrange()? in a string are uppercase?

range(): Returns a list in Python 2 and a range object in Python 3.
xrange(): Only available in Python 2, it returns an xrange object that generates numbers on the fly (memory-efficient).

29. Name some immutable types in Python.

Tuple
String
Number (int, float)

30. Name some mutable types in Python.

List
Set
Dictionary

31. What is a lambda function in Python?

A lambda function is a small, anonymous function that can take multiple arguments but only contains a single expression.

square = lambda x: x * x
print(square(4))

Output: 16

32. How do you write a lambda function in Python?

add = lambda x, y: x + y
print(add(3, 7))

Output: 10

33. How many types of loops are available in Python?

Python supports two types of loops:

for loop
while loop

34. What will the following code output?

numbers = [1, 2, 3, 4, 5]
print(numbers[:-3])

Output: [1, 2]

35. What will the following code output?

numbers = [1, 2, 3, 4, 5]
print(numbers[:])

Output: [1, 2, 3, 4, 5]

36. What are Python modules?

Python modules are files containing Python code (functions, classes, or variables) that can be reused in other Python programs.

37 Name some common built-in Python modules.

os
sys
math
random
json

38. How do you import a specific attribute from a module?

Use the from module import attribute syntax.

from math import sqrt
print(sqrt(16))

Output: 4.0

39. Can you write Python code without indentation?

No, Python strictly enforces indentation to define code blocks. Without it, the code will throw an IndentationError.

40. What’s the difference between Python arrays and lists?

Arrays: Can only store elements of the same data type.
Lists: Can store elements of different data types.

import array as arr
numbers = arr.array("i", [1, 2, 3])

41. How do you create an array in Python?

Use the array module to create an array.

import array as arr
my_array = arr.array("i", [1, 2, 3])
print(my_array)

Output: array(‘i’, [1, 2, 3])

42. What will be the output of the following code?

s = {1, 1, 1, 1}
print(s)

Output: {1}

Explanation: Sets in Python do not allow duplicate values.

43. Can you create an empty set using {}?

No, using {} creates an empty dictionary. To create an empty set, use the set() function:

empty_set = set()
print(type(empty_set))

Output: <class 'set'>

44. What is the purpose of the set() function?

The set() function creates a set, which is an unordered collection of unique elements.

45. Which Python collection is used for operations like union, intersection, and difference?

Sets are used for these operations.

a = {1, 2, 3}
b = {3, 4, 5}
print(a.union(b))  # Output: {1, 2, 3, 4, 5}

Output: {1, 2, 3, 4, 5}

46. Does Python support object-oriented programming?

Yes, Python supports object-oriented programming with features like classes, inheritance, and polymorphism.

47. What is init in Python?

class Person:
    def __init__(self, name):
        self.name = name

48. What is the purpose of the self keyword in Python?

The self keyword represents the instance of a class. It is used to access attributes and methods within the class.

49. Is it necessary to use self in a class?

Yes, it is necessary to use self in instance methods to refer to the current object.

50. What is a class attribute?

A class attribute is shared across all instances of a class. It is defined outside the methods and belongs to the class rather than an instance.

51. Does Python support inheritance?

Yes, Python supports inheritance, including single, multiple, and multilevel inheritance.

52. What is the purpose of the super() method?

The super() method allows access to methods in a parent class.

class Parent:
    def greet(self):
        print("Hello from Parent")

class Child(Parent):
    def greet(self):
        super().greet()
        print("Hello from Child")

53. What does the is operator do in Python?

The is operator checks if two variables point to the same object in memory.

a = [1, 2]
b = [1, 2]
print(a is b)

Output: False

54 How do you work with dates and times in Python?

Use the datetime module.

import datetime
now = datetime.datetime.now()
print(now)

Output: It should print the current date and time.

55. What does it mean that Python is dynamically typed?

It means variables in Python do not need explicit declaration of their type. The type is determined during runtime.

56. Are Python functions first-class objects?

Yes, Python functions are first-class objects, meaning they can be assigned to variables, passed as arguments, and returned from other functions.

57. How do you reverse an array in Python?

import array as arr
my_array = arr.array('i', [1, 2, 3, 4])
print(my_array[::-1])

Output: array(‘i’, [4, 3, 2, 1])

58. What are *args used for in Python?

The *args syntax allows a function to accept a variable number of positional arguments.

def greet(*names):
    for name in names:
        print(f"Hello, {name}")
greet("Alice", "Bob", "Charlie")

59. How do you split a string into a list in Python?

Use the split() method.

text = "Python is great"
words = text.split()
print(words)  # Output: ['Python', 'is', 'great']

Output: [‘Python’, ‘is’, ‘great’]

60. How do you check if a given string is a palindrome?

def is_palindrome(s):
    return s == s[::-1]

print(is_palindrome("madam"))
print(is_palindrome("hello"))

Output: It should print True for the first print() statement and False for the second.

61. How do you remove vowels from a string in Python?

def remove_vowels(s):
    vowels = "aeiouAEIOU"
    return ''.join([char for char in s if char not in vowels])

print(remove_vowels("Python"))

Output: Pythn

62. Can you pass one function as an argument to another function in Python?

Yes, since Python functions are first-class objects, you can pass one function as an argument to another.

63. What are docstrings in Python?

Docstrings are string literals used to document a function, class, or module. They are written using triple quotes.

def greet():
    """This function greets the user."""
    print("Hello!")

64. How can you access a function’s docstring?

Use the __doc__ attribute.

def greet():
    """This function greets the user."""
    pass

print(greet.__doc__)

Output: This function greets the user.

65. What’s the difference between a comment and a docstring?

Comments are ignored by Python and cannot be accessed during runtime.
Docstrings are used to document code and can be accessed using the __doc__ attribute.

66. Does Python support multiple inheritance?

Yes, Python supports multiple inheritance.

class A:
    pass

class B:
    pass

class C(A, B):
    pass

67. How do you create an empty class in Python?

Use the pass statement:

class EmptyClass:
    pass

68. What are the different types of functions in Python?

Built-in functions (e.g., len(), print())
User-defined functions

69. What types of operators are available in Python?

Arithmetic operators
Logical operators
Comparison operators
Bitwise operators
Assignment operators
Identity operators

70. Where can Python be used?

Python is used in various domains, including:

Web development
Data science
Machine learning
Artificial intelligence
Game development

71. What is Django?

Django is a high-level Python web framework that enables rapid development of secure and maintainable websites. Not sure whether Django or Flask fits your project? Read Django vs. Flask.

72. What are some key features of Django?

Built-in authentication system
URL routing
ORM (Object-Relational Mapper)
Template engine
Built-in admin interface

73. Can Django be used for backend development?

Yes, Django is primarily used for backend development, offering powerful tools for handling server-side logic and database operations.

74. Name some databases supported by Django.

SQLite
MySQL
PostgreSQL
Oracle

75. What architectural pattern does Django follow?

Django follows the Model-View-Template (MVT) architectural pattern.

76. What are the advantages of using Django?

Easy to learn and use.
Secure by default.
Scalable for large applications.
Extensive community support.
Reduces development time.

77. What are the disadvantages of using Django?

Overhead for small applications.
Limited flexibility in ORM.
Can be bulky for simple projects.

78. What are Django signals?

Django signals allow certain senders to notify listeners when specific events occur.

79. Name two key parameters used in Django signals.

Sender: The model or function that sends the signal.
Receiver: The function or method that listens to the signal.

80. What is a Django session?

A Django session allows storing user-specific data, such as login information, on the server side.

81. What is the role of Django views?

Django views handle user requests, process data from models, and return responses to templates for rendering.

82. How do Django views work with models and templates?

Models: Fetch data from the database.
Views: Process the data and pass it to templates.
Templates: Display the processed data in a user-friendly format.

83. What is Flask?

Flask is a microframework written in Python that is lightweight and ideal for small applications or prototyping.

84. How can Flask send emails?

Flask can send emails using the flask-mail extension, which integrates email-sending functionality.

85. Why is Flask called a microframework?

Flask is termed a microframework because it provides only core functionalities, such as routing and request handling, while additional features are implemented via extensions

86. How do you add ORM functionality to Flask?

Use the Flask-SQLAlchemy extension to integrate an ORM with Flask applications.

87. What are the advantages of using Flask?

Lightweight and flexible
Easy to learn for beginners
Supports extensions for added functionality
Comes with a built-in development server

88. How do you differentiate between Django and Flask?

Django: A full-stack framework suited for large applications.
Flask: A microframework ideal for smaller applications and prototyping.

89. What are iterators in Python?

Iterators are objects in Python that allow traversing through elements, one at a time, using the __next__() method.

90. Which method is used to create custom iterators in Python?

The __iter__() method is used to initialize the iterator, and the __next__() method is used to iterate through elements.

91. What happens at the end of an iterator?

A StopIteration exception is raised when there are no more elements to traverse.

92. What are Python generators?

Generators are a special type of iterator that allows generating items lazily using the yield keyword instead of storing them in memory.

93. Write a simple Python generator example.

def generate_numbers():
    for i in range(3):
        yield i

for number in generate_numbers():
    print(number)

94. What is the difference between yield and return?

return: Terminates the function and returns a value.
yield: Pauses the function, saving its state, and resumes on the next call.

95. Do generators automatically implement iter() and next()?

Yes, Python generators automatically implement these methods, making them easier to use than custom iterators.

96. Can you create a generator without using yield?

No, the yield keyword is mandatory for defining a generator.

97. What are Python decorators?

Decorators are functions in Python that modify the behavior of another function or method. They are often used for logging, validation, or access control.

98. Write an example of a Python decorator.

def decorator(func):
    def wrapper():
        print("Before function call")
        func()
        print("After function call")
    return wrapper

@decorator
def say_hello():
    print("Hello!")

say_hello()

99. How do you use regular expressions in Python?

Import the re module to work with regular expressions.

import re
pattern = r"\d+"
match = re.search(pattern, "There are 123 apples")
print(match.group())  # Output: 123

Output: 123

100. What does the * signify in regular expressions?

The * matches zero or more occurrences of the preceding character or group.

Congrats if you made it this far in the guide! We are almost there. ;)

101. Is Python 2 still supported?

No, Python 2 reached its end of life on January 1, 2020, and is no longer officially supported.

102. What is PEP 8?

PEP 8 is the official Python style guide that provides conventions for writing clean, readable code.

103. How does Python manage memory?

Python manages memory using:

A private heap space for all objects and data structures.
An in-built garbage collector to reclaim unused memory.

104. What is slicing in Python?

Slicing allows you to extract a portion of a sequence (e.g., list, string) using a specified range of indices.

my_list = [0, 1, 2, 3, 4]
print(my_list[1:4])  # Output: [1, 2, 3]

Output: [1, 2, 3]

105. How do you copy an object in Python?

Use the copy() or deepcopy() method from the copy module.

106. What is the Pyramid framework used for?

Pyramid is a Python web framework designed for building large, complex web applications.

107. What are the limitations of Python?

Slower compared to compiled languages like C++.
Limited for mobile development.
GIL (Global Interpreter Lock) can be a bottleneck for multithreaded applications.

108. How do you remove duplicate values from a list in Python?

my_list = [1, 2, 2, 3, 4, 4]
unique_list = list(set(my_list))
print(unique_list)

Output: [1, 2, 3, 4]

110. What is the purpose of the ** operator?

The ** operator is used for exponentiation.

print(2 ** 3)

Output: 8

110. What are membership operators in Python?

in: Checks if a value exists in a sequence. -
not in: Checks if a value does not exist in a sequence.

111. What is raw_input() in Python?

raw_input() is used in Python 2 to take user input as a string. In Python 3, it has been replaced with input().

112. Which Python collection is most similar to JavaScript objects?

A dictionary in Python is similar to JavaScript objects, as both store key-value pairs.

113. Why is Python often recommended for beginners?

Simple and readable syntax.
Extensive standard library.
Versatile for various use cases.
Dynamically typed.

114. What’s the difference between Python and Java?

Typing: Python is dynamically typed, while Java is statically typed.
Syntax: Python uses indentation for blocks, while Java uses curly braces {}.
Speed: Java is faster due to its compiled nature, whereas Python is interpreted.

115. Explain the following code snippet.

a, b, c = 1, 2, 3

Answer: The code initializes three variables a, b, and c with the values 1, 2, and 3, respectively.

116. What does the following code do?

a = b = c = 1

Answer: All three variables a, b, and c are assigned the same value of 1.

117. What is the with statement in Python used for?

The with statement is used for resource management, such as opening files. It ensures that resources are properly released, even if an error occurs.

with open("file.txt", "r") as file:
    content = file.read()

118. What does the global keyword do?

The global keyword allows modification of a variable declared in the global scope from within a function.

x = 5

def modify_global():
    global x
    x = 10

modify_global()
print(x)

Output: 10

119. What will the following code output?

print(['Python'] * 2)

Answer: [‘Python’, ‘Python’]

Explanation: The multiplication operator repeats the elements of the list.

120. What is pickling in Python?

Pickling is the process of serializing a Python object into a byte stream using the pickle module.

import pickle

data = {"key": "value"}
with open("data.pkl", "wb") as file:
    pickle.dump(data, file)

121. What is unpickling?

Unpickling is the reverse process of converting a byte stream back into a Python object using the pickle module.

with open("data.pkl", "rb") as file:
    data = pickle.load(file)
print(data)

Output: {‘key’: ‘value’}

122. Does Python pass arguments by value or reference?

Python uses pass-by-object-reference, meaning that mutable objects can be modified inside a function, while immutable objects cannot.

123. Is Python case-sensitive?

Yes, Python is case-sensitive, meaning variable names like Var and var are treated as distinct.

125. What will be the output of this code?

l = [1, 2, 3, 4, 5]
print(l[10:])

Answer: An empty list []

Explanation: Slicing a list beyond its range does not raise an error but returns an empty list.

126. What types of databases are supported by Python?

Python supports both SQL and NoSQL databases, including:

SQL: MySQL, PostgreSQL, SQLite
NoSQL: MongoDB, Cassandra

127. Can Python code written on Windows run on Linux?

Yes, Python code is cross-platform and can run on different operating systems without modification, provided the required dependencies are available.

128. What happens when you execute the following code?

f = open("new.txt", "w")

Answer: It opens the file new.txt for writing. If the file does not exist, it creates it. If it exists, its content is overwritten. To learn more about file modes, see reading and writing files in Python.

129. When is the ZeroDivisionError exception raised?

This exception is raised when a division or modulo operation has a denominator of zero.

130. What’s the difference between abs() and math.fabs()?

abs(): A built-in function that works with integers, floats, and complex numbers.
math.fabs(): Part of the math module and works only with floats.

131. What does the -1 index signify in a sequence?

l = [10, 20, 30]
print(l[-1])

Output: 30

132. What is a Python module?

A Python module is a file containing Python code (functions, classes, or variables) that can be reused in other programs.

133. How do you import a module in Python?

Use the import statement:

import math
print(math.sqrt(16))

Output: 4.0

134. Can you create custom modules in Python?

Yes, you can create a custom module by saving Python code in a .py file and importing it into other scripts.

135. What is the dir() function used for?

The dir() function is used to display the attributes and methods of an object.

136. How do you check the type of a variable in Python?

Use the type() function:

x = 10
print(type(x))

Output: <class 'int'>

137. How do you handle exceptions in Python?

Use try-except blocks:

try:
    result = 10 / 0
except ZeroDivisionError:
    print("Cannot divide by zero")

138. What is the difference between is and == in Python?

is: Checks if two variables point to the same object in memory. -
==: Checks if two variables have the same value.

139. What is Python’s GIL?

The Global Interpreter Lock (GIL) is a mutex in CPython that allows only one thread to execute Python bytecode at a time, limiting multi-threading capabilities.

140. How do you create a virtual environment in Python?

Use the venv module:

python -m venv myenv

Activate it using:

Windows: myenv\Scripts\activate
Linux/Mac: source myenv/bin/activate

141. How do you list all installed Python packages?

Use the pip list command in the terminal to display all installed Python packages and their versions.

pip list

142. How do you install a specific version of a Python package?

Use the pip install command with the package name and version:

pip install package_name==1.2.3

143. What is a namespace in Python?

A namespace is a container that holds identifiers (variable names, function names, etc.) and maps them to their corresponding objects.

144. What are Python’s built-in data types?

Numeric types: int, float, complex
Sequence types: list, tuple, range
Text type: str
Mapping type: dict
Set types: set, frozenset
Boolean type: bool

145. What is the purpose of the enumerate() function in Python?

The enumerate() function adds a counter to an iterable and returns it as an enumerate object.

colors = ["red", "blue", "green"]
for index, color in enumerate(colors):
    print(index, color)

Output: 0 red 1 blue 2 green

146. What is the difference between deepcopy() and copy()?

copy(): Creates a shallow copy of an object, which copies the object but not nested objects.
deepcopy(): Creates a deep copy of an object, including all nested objects.

147. What are Python’s file handling modes?

“r”: Read mode
“w”: Write mode (overwrites file)
“x”: Create mode (fails if the file exists)
“a”: Append mode
“b”: Binary mode
“t”: Text mode

148. What is a Python metaclass?

A metaclass is a class of a class. It defines how classes behave and can be used to control class creation.

149. How do you create a thread in Python?

Use the threading module:

import threading

def task():
    print("Thread is running")

t = threading.Thread(target=task)
t.start()

150. What are f-strings in Python?

F-strings, introduced in Python 3.6, provide a concise and efficient way to format strings.

name = "Alice"
age = 25
print(f"My name is {name} and I am {age} years old.")

Conclusion

Phew! That was a lot to digest I assume. But you made it to the end.

Preparing for an interview can feel overwhelming. And with these 150 Python interview questions. you have a solid resource to help you with the most commonly asked Python interview questions.

Remember, to practice each question thoroughly. I also have other guides and tutorials on my website, so feel free to go over them to help you better understand some of the concepts.

What other Python interview questions can you think of?

Django vs. Flask: Choosing the Right Web Framework

Thu, 31 Aug 2023 00:00:00 GMT

Python programming language has gained immense popularity in the last two decades.

Today, Python is not only one of the most widely used programming languages but also one of the most loved programming languages.

There are many reasons why Python wins over other programming languages. And one of the reasons is that it is beginner-friendly.

That’s why its use is increasing in web development, machine learning, game development, and data science.

Web development is one of the essential domains where you can use Python.

Python provides several web frameworks that are very useful in developing web applications. For example, some popular Python web frameworks are Django, Pyramid, Flask, Grok, TurboGears, and CherryPy.

However, we will not discuss any framework other than Django vs. Flask.

Django and Flask are the two most popular Python web frameworks.

Both frameworks are helpful to web developers worldwide in developing modern web applications.

But people are often confused about choosing between these two frameworks. Also not know how they differ. There are several differences between them, and one should choose carefully.

This article will compare these two frameworks and discuss their pros and cons. Ultimately, we will see which of them is better and when to choose Django or Flask.

Origin

Django is the most famous name in the Python web development community. It is an open-source, free Python web framework based on MTV architectural patterns.

Returning to its origin, Django was created in 2003 by Adrian Holovaty and Simon Willison. And then it officially came out in 2005.

The main aim of creating Django was to simplify web development because, at that time, JavaScript had its limitations.

There was still time for popular JavaScript frameworks such as Angular and React to emerge. So Python, being a popular language, was used to create Django. Its purpose was to simplify the whole web application development process. So the creators decided to focus more on reusing components, rapid growth, and less code.

Earlier frameworks such as Plone and Zope influenced Django. And Django’s success inspired other popular Python frameworks such as CherryPy, Pyramid, and Bottle.

Unlike Django, Flask did not have a long-term purpose. Instead, it started as an April fool joke and had a surprise beginning. Armin Ronacher created two different solutions.

One was Werkzeug, a server framework, and Jinja2, a template library. He tried an experiment just for fun.

He added these two solutions in a zip file and made it part of a framework he named denied framework. Armin worked on this framework when his earlier answers became part of it. When another developer installed the restricted framework, the solutions unzipped automatically.

When these two solutions ran together with the denied framework, to surprise, it received a positive framework. Thus, Armin created this new framework which is now known as Flask.

Django vs. Flask

Both Django and Flask are Python frameworks. But Django is a full-fledged framework while Flask is a microframework. Simply put, Flask provides the minimum features required to run the server and does not have additional features.

On the other hand, Django is a high-level full-stack framework that provides several other features, including database layers. In simple words, with Django, almost everything present that is required. On the other hand, you can integrate additional features in Flask.

Django follows the Model-View-Template architectural pattern.

As a result, the design pattern helps build web applications more efficiently in less time and with less code. But Flask only comes with essential tools that help build basic web applications, which is one of the main differences between Django and Flask.

Working with databases

Perhaps, one of the essential differences between Django and Flask is the way both of them work with databases. A database is an integral part of a web application.

Databases are one of the main focuses of backend development. Thus, a framework needs to be efficient.

Django has a Relational Object Mapping known as the Django ORM.

Some of its features allow the Django ORM to work with databases such as MySQL, SQLite, and Prophet.

It does not require long, confusing SQL queries. Moreover, its ORM uses data models that are useful in linking databases with classes.

Flask neither has ORM nor data models.

On the hand, developers need third-party toolkits to perform everyday database operations with Flask.

Ease of learning

Both Django and Flask are Python web frameworks. Hence, it is critical to understand the Python programming language well.

Both can be integral to web development, so the concepts are more or less similar. But, as mentioned earlier, Django is a full-stack framework, and it offers many ideas, and it takes time to grab them.

Flask, on the other hand, is a microframework that has fewer concepts as compared to Django. If you have proper Python and basic HTML knowledge, learning Flask will not take much time.

But Flask requires additional extensions. So to work correctly with it, you need to understand them.

Packages

Django comes with a wide range of built-in packages. There are over four thousand packages in Django, and thus it makes it easier to implement complex functionalities in an application.

In comparison to Django, Flask does not offer such packages. Instead, it is a minimalistic framework that allows us to choose third-party packages. However, Flask is more flexible and extensible than Django.

Scalability

Django MVT architectural pattern makes it very powerful and scalable. As mentioned earlier, Django supports a variety of databases because of its powerful ORM. Moreover, its data models make it more efficient to work with the backend. Web applications built using Django are much faster and more scalable than Flask.

However, Flask does not have Django-like architecture and ORM.

Security

Being secure is an essential feature of a web framework. Today, security is the top priority of every web application.

Django has different measures for different attacks. For example, templates in Django are capable of protecting from XSS attacks. In addition, it has built-in protection from CSRF attacks and measures for SQL injections.

Flask also has a security library called Flask-security library that provides security benefits over such attacks. But it is not as efficient as Django.

Popularity

Both of these frameworks are most popular in the Python web development community.

According to StackOverflow’s developer survey 2020, both Django and Flask had received the same stats (i.e., 14.2%) in the list of most popular web frameworks by all respondents.

But in the same list of professional developers, Flask received 13.6% while Django received 13.2%.

And again, in the same survey, Django got 4th position in the list of “most wanted web frameworks,” while Flask got 7th.

So no doubt, both of them are popular. But Django is more in demand than Flask.

Community support

Django is older than Flask. As it is an open-source product, its community support has increased tremendously. As a result, it has enormous community support, while Flask’s community is not significant.

When to use Django and Flask?

So, where should we use Django and Flask? Both are Python frameworks used to develop web applications and share some common concepts, but they are pretty different in some aspects.

Django is a full-fledged high-level full-stack web framework.

As discussed, it provides almost everything required to build web applications. It comes with an ORM and data models that are very helpful with backend development.

Moreover, it has thousands of built-in packages, and because of these packages, there is no need to look elsewhere.

Because of such features and its MVT architectural pattern, Django is powerful enough to develop large-scale web applications.

Some of the most popular applications that use Django are:

Instagram
Spotify
DropBox
Mozilla
Washington Post
National Geographic
Pinterest
NASA

As you can see, these applications are complex. And their everyday traffic is enormous. Not only that but also these applications has to integrate advanced security features. And as Django is robust, secure, high-performing, and scalable, it meets many complex requirements.

On the other hand, Flask does not have some heavy-duty features such as Django. But that does not mean the giants do not use Flask.

Because of its simplicity, extensibility, and, most importantly, flexibility, Flask is helpful to developers who want more room for development.

Some of the popular applications that use Flask are:

Netflix
Airbnb
Red Hat
Reddit
Lyft

So, if you want to create a medium to large-sized application and you want everything in a single place, then Django is a good choice. I

Whereas, if you want to make a simple, small-sized application with flexibility, room for developers, and custom components, then Flask is a good choice.

Django or Flask?

So, both frameworks help build web applications. That’s why choosing one of them for your project depends on your requirements.

If the project is large and requires powerful features, then Django is a perfect option.

But if the project is small and needs only core features, Flask is a good option.

Also, a developer’s perspective is essential. If you want flexibility, then Flask is better than Django. Moreover, choose Django instead of Flask if you want everything in one place.

But the size of the project also comes into play. There is a good chance Flask can get the job done for smaller projects. So, no need to choose a bulky framework like Django.

However, if the project is medium-sized to large, go for Django.

Conclusion

This article focuses on the two most popular Python web frameworks - Django vs. Flask.

We discussed how the processes behind their development and their basis.

In addition, we discussed their main differences and features. We also discussed where Django and Flask are useful and how to make decisions reflecting on the purpose of your projects.

Let’s discuss their differences in brief:

Django came out in 2005, while Flask came out in 2010.
Django is a full-fledged framework with several features, such as ORM, data models, and packages. Flask is a lightweight micro-framework.
Flask is simple and beginner-friendly. In comparison, Django is huge and requires experience.
Django’s community is vast. And Flask’s community is still growing.
Flask is a minimalistic framework that only provides core features. At the same time, Django has numerous amount of packages available to use.
In short, Django is a good option for medium and large-sized projects. And in comparison, Flask is efficient for smaller-sized projects.

Both Django and Flask are very popular.

Because of that, they are well known among developers for building web applications. Furthermore, their popularity is increasing exponentially. Not only that, but they also have a perfect place in the job market.

Want to test what you know about Django? The Python interview questions guide includes a few Django basics.

An Essential Guide to Python Text Editing & Processing

Mon, 17 Jul 2023 00:00:00 GMT

As a programmer, a large part of my work may involve working with texts & Strings. That’s why it is crucial to know some of the fun aspects of Python text editing & processing using the standard library.

As a result, I thought about writing this tutorial for those who want to get their hands dirty with Python text editing & processing.

Here’s what we will cover:

Work on three basic String operations through the standard library.
Learn how to search Strings for specific information.
Understand how to parse and manipulate strings.
Learn how to format Strings.

Before I move further, you already need to be familiar with Python. Moreover, I highly recommend that you have the basics down first before jumping into Python text editing & processing.

So without a further due, let’s get started.

Basic String Operations

In the first part of the Python text editing & processing article, we will learn how to perform basic String operations.

One of the powerful features of the Python standard library is that it provides essential tools for testing and filtering String content.

Go ahead and open your IDE to import the string module.

import string

Let’s see some of the pre-defined String constants the string module provides. And these are:

print(string.ascii_letters)
print(string.digits)
print(string.hexdigits)
print(string.punctuation)

Output:

abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ
0123456789
0123456789abcdefABCDEF
!"#$%&'()*+,-./:;<=>?@[\]^_`{|}~

Here we used the string module to print out some of the constants that the standard library provides. And as a result, you can see we have some values such as letters, digits, and even hex values.

Moving on, let’s create three variables and assign some values or Strings to them.

stringOne = 'It is raining cats and dogs.'
stringTwo = 'Labratory'
stringThree = '01234567'

Moreover, I also want to check if a String is alphanumeric or not. For that, we can use the alnum() function. Here’s how it works:

print(stringOne.isalnum())
print(stringTwo.isalnum())
print(stringThree.isalnum())

Output:

False
True
True

The outputs are all boolean values. The first result shows that our variable stringOne is not alphanumeric since there are spaces within the Strings. On the other hand, stringTwo & stringThree came out to be alphanumeric since they meet the condition of having alphabets or numbers.

Similarly, we can also check if a String only contains alphabets. To do that, we can use the isalpha() function. Let’s check it on the stringOne.

print(stringOne.isalpha())

Output:

False

The output is False because stringOne has spaces between the words.

Spaces are not considered alphabets by Python. As a result, when you run the function isalpha() on stringOne, we get False.

Next, we can also verify if a String is numeric or not using the isnumeric() function.

And this is how it works:

print(stringThree.isnumeric())

Output:

True

Let’s run this function on stringTwo:

print(stringTwo.isnumeric())

Output:

False

As you can see, that the outputs are True and False.

True because stringThree has numeric characters, while the other one is False since that has no numeric values.

Searching Strings

We can also search Strings for specific data. Fortunately, the Python String object comes with built-in methods to do this.

First, let’s see how we can use the startswith() & endswith() functions to check if the String starts or ends with a specific sequence of characters.

Go ahead and type the following:

sampleString = 'The sky is blue'

Now I want to check whether our sampleString has ‘the’ in it. Thus, I can type:

print(sampleString.startswith('The'))

Output:

True

We get True since the word is present.

Okay, now what if we passed the lowercase version of that word.

Type the following:

print(sampleString.startswith('the'))

Output:

False

So, these functions are case-sensitive. Since lowercase ‘the’ is not present, we are getting False.

Lastly, let’s try the endswith() function.

print(sampleString.endswith('blue'))

Output:

True

We get True because we can see that the word ‘blue’ is at the end of our phrase.

Apart from the startswith() and endswith() functions, there are also other general functions such as find() and rfind().

You can use these two functions for finding data anywhere in the String.

First, let’s see the find() function in action. We can try it on the sampleString that we created earlier.

print(sampleString.find('sky'))

Output:

For example, we passed the sky. This String exists at index position 4.

Additionally, you can also check for sub-String using the in operator:

print('blue' in sampleString)

Output:

True

As a result, we get True since the word blue is present within the sampleString.

Moving on, let’s talk about the replace() function to replace contents within a String.

Let’s try it on our sampleSring. For example:

print(sampleString.replace('blue','black'))

Output:

The sky is black

So, I used the function to replace the word blue with black.

Before I end this section, I also want to show you the count() function.

This function allows us to count the instances of a sub-String and returns the number of times it occurs.

I am going to create a new String and use the count() function on that.

newString = 'I can see a car parked outside our house. It is a red car.'

Now let’s see the count() function in action:

print(newString.count('car'))

Output:

And as you can see that we get two since the word car occurs twice. Also, remember that these functions are all case-sensitive. If I pass the word car with an upper case C, then the result would be 0. Since no words are within our String, starting with an upper case C.

String Manipulation

An essential concept that involves Python text editing & processing is the ability to manipulate String.

First, let’s start by defining a new sample String. So, I will type:

newString = 'It is a sunny day in Southern California.'

To convert all the letters into upper case, we can use the upper() method.

print(newString.upper())

Output:

IT IS A SUNNY DAY IN SOUTHERN CALIFORNIA.

The same goes with converting all the letters to lower case. We will use the lower() method.

print(newString.lower())

Output:

it is a sunny day in southern california.

Now let’s talk about the split() function, which you can use to break up a single text String into multiple Strings. For instance, suppose we want to split each of the words within our newString into an array based on spaces between them.

Here’s what I mean. Type the following:

splitString = newString.split(" ")
print(splitString)

Output:

['It', 'is', 'a', 'sunny', 'day', 'in', 'Southern', 'California.']

As I ran that, you can see that we have split the String into an array.

What if you want to join all the words inside your array? For that, we can use the join() function. It does the exact opposite.

The join() function will take the individual Strings and convert them into one single String.

For example:

joinString = ' '.join(splitString)
print(joinString)

Output:

It is a sunny day in Southern California.

Now we have successfully converted the splitString array and joined them together again.

We joined each String into one single String, with the spaces between them being the separators.

On the other hand, if you don’t want any space between the words, then type this:

joinString = ''.join(splitString)
print(joinString)

Output:

ItisasunnydayinSouthernCalifornia.

As a result, the function went with attaching all the words since we passed no separators.

Python has a feature that allows us to perform batch replacement of characters in a String through a translation table.

Here’s what I mean.

First, let’s create a translation table with the maketrans() function.

transTable = str.maketrans('abcdef','123456')

So now we have our translation table defined as transTable with some characters that we want to replace with some numbers. For instance, the letter a will become 1, and b will become 2, and so on

Then go ahead and define a new variable with the following:

myString = 'This is a big deal for California.'
print(myString)

Output:

This is a big deal for California.

Now let’s go ahead and use the translation table transTable to perform the translate operation. To do that, we’ll use the translate() function.

So type:

print(myString.translate(transTable))

Output:

This is 1 2ig 451l 6or C1li6orni1.

You can see that some of the characters have now become numbers we defined in our translation table transTable earlier.

To sum up, batch replacement in Python allows us to define translation tables that we can use to replace characters with some other characters that we specify.

In the next part of the Python text editing & processing tutorial, we learn how to work with String formatting.

String Formatting

In this part of the Python text editing & processing tutorial, we will take a good look into String formatting.

Above all, we will solely focus on formatting Strings using the template Strings and the format() function.

Now, let me create and define a new variable. We can call it newString2.

newString2 = 'It is a $weather day in the $direction Coast.'

I will explain in a bit why I am using the dollar signs.

Okay, the first concept in this part is how to use template Strings to perform String formatting.

I have to import the class Template from the String module. Here’s how you do it:

from string import Template

Now you can see that newString2 has two dollar signs which shows that these are placeholders or variables.

We can substitute these placeholders with Strings or values. There are usually two steps involved to do this.

First, we create an instance of the Template class:

theInstance = Template(newString2)

Then you have to call the substitute() function on the template and pass the placeholders with values.

Here’s how it works:

outputString = theInstance.substitute(weather='Rainy',direction='East')
print(outputString)

Output:

It is a Rainy day in the East Coast.

The output shows that we have actual values rainy & East replacing the placeholders we defined before using the dollar sign.

Let’s move on to learning how to use the format() function. First, I will start by creating two variables as follows:

fruit = 'apples'
color = 'red'

So what we are going to do is substitute these two values into a new String using the format() function.

And this is how it works:

newStr = 'Those {} are {}.'.format(fruit,color)
print(newStr)

Output:

Those apples are red.

We can see that the curly braces acted as placeholders. After that, we used the format() function and passed the names of the variables fruit and color.

In short, it took the two variables in order and put their values into the String where we have the curly braces.

In addition, you can also substitute named variables. Here’s how it works:

print('Those {var1} are {var2}.'.format(var1=fruit,var2=color))

Output:

Those apples are red.

The variables var1 and var2 are substituted.

Replacing & Removing Characters from a String

One of the fastest ways to replace a character with a new character in Python is by using the replace() function.

For example, we have the following String:

myStr = 'abcdefghijk12345'

I want to replace the first character a with the letter z, and here’s how I can do it:

print(myStr.replace('a','z'))

Output:

zbcdefghijk12345

The replace() function returns a new String, while the original String remains unchanged. If you want the change to be permanent, then you can do this:

myStr = myStr.replace('a','z')
print(myStr)

Output:

zbcdefghijk12345

Now the change is permanent for myStr.

The replace() function can also remove characters if we don’t pass any values as the second parameter.

For example:

print(myStr.replace('z',''))

Output:

bcdefghijk12345

As a result, we can now see that the letter z is gone from our String.

Lastly, the replace() can also remove spaces from a String.

Let’s say I have the following String:

newStr = 'a b c d e f g h'

You can see that we have extra spaces between the characters, so I want to get rid of them.

For that I can type:

print(newStr.replace(' ',''))

Output:

abcdefgh

Moreover, to make the change permanent:

newStr = newStr.replace(' ','')
print(newStr)

Output:

abcdefgh

As a result, the change is now permanent for newStr.

Conclusion

This section brings us to the end of Python text editing and processing. We went over some of the crucial concepts that you will need to know if you want to edit, process, and manipulate text with Python.

Depending on your situation, you may have to search Strings, format Strings, remove characters from a String or even replace them with new values. When it comes to Python text editing & processing, the functionality of Python can be endless.

Most importantly, if you work with big data or need to parse specific characters or sub-String, Python text editing and processing is an efficient way to handle those data.

Once you can handle text in memory, reading and writing files in Python shows you how to work with it on disk.

The text editing & processing functionality of Python is pretty cool. What other programming languages do you think are also suited for working with text?

KNN Algorithm Using Scikit-Learn: Classifying Iris Species

Wed, 22 Feb 2023 00:00:00 GMT

KNN Algorithm (K-Nearest Neighbor) is a supervised learning classification algorithm to perform predictive modeling and classify data.

The KNN algorithm learns from example data and able to classify new data based on feature similarity.

The primary function of supervised learning algorithms is to learn from labeled data points with some features to identify unlabeled data.

Think of labeled data as training data or examples that already come with the correct answer or classification. Therefore, this is the data that you feed into the model for training.

And, you feed the test data to test and evaluate your model’s prediction.

The purpose of this article is to get your hands dirty with the KNN algorithm by building an actual machine learning model. The model that we will create is going to classify different species of iris.

How Does KNN Algorithm Work?

The KNN algorithm uses the concept of similarity. It assumes that similar objects or data points exist near to each other.

Imagine plotting your data as points on a graph. Similar data points end up close to each other. Let’s say the blue points are cats. And the red points are the dogs. Let’s assume that this data is inputted based on features such as weights and heights.

As a result, points with similar features are clustered close to each other.

Now imagine a few green data points. They represent animals that we need to classify and predict, whether we have cats or dogs. In this case, you only have the information on their weights and heights.

Based on their features we are able to associate them with either the neighboring blue points or the red points.

For example, imagine a green point that sits close to the red points. This data point is most likely to be a dog.

Now to make the prediction, that’s where the KNN algorithm will come into play.

KNN will calculate the distance between a new data point with every single data point from your training set.

It converts the data points into mathematical values or vectors, which it uses to find the distances.

One of the most popular distance measuring metrics is the Euclidean distance.

Euclidean distance works by calculating the distance between two data points in a plane.

For instance, if we want to calculate the Euclidean distance between two data points p & q, we can use the following formula:

d(p, q) = √( (q₁ − p₁)² + (q₂ − p₂)² + … + (qₙ − pₙ)² )

Once the algorithm calculates every single distance, it selects the K nearest data points.

What do I mean by K nearest?

K is the number of nearest neighbors or labeled data points in proximity to your new data point that we are trying to classify. Most importantly, the value of K is a positive integer.

In other words, if we have K=4, then the algorithm looks for the four nearest data points around the unlabeled point.

Let’s assume that the value of K for the unknown green data point is 3. Thus, we will draw a circle around the green data point with three neighbors inside the proximity.

Out of 3 of the neighboring data points, 2 of the data points are red, and the other is blue. In other words, we have two nearest points that belong to the red class and one nearest point belonging to the blue.

For example, the majority of the neighbors inside the circle are red. And as a result, our algorithm will most likely predict that our new data point belongs to the red class.

However, it probably is not going to be this simple. This is an example using imaginary data points. But hopefully, you get the idea of how our algorithm uses the distance and the K value to make predictions.

It is crucial to pick the optimal value for K to minimize the number of biases and variances. Thus, a K value should neither be too small nor too large.

And the best way to find out the optimal K value is by testing your model with different K values.

To sum up, here’s how we implement the KNN algorithm:

Load and store the data.
Calculate the distance from x (new data point) to all other data points.
Sort all the distances from your data in ascending order.
Initialize the K value for the nearest data points.
Make a prediction based on the majority of data points with the same label within the K value.
Evaluate your machine learning model.

Before we jump into the new section, I am assuming you are already familiar with the following concepts:

Python
NumPy
Pandas
Matplotlib
Supervised Machine Learning
Scikit-Learn (Optional)

Classifying Iris Species

Moving onto the fun part where we will now build an actual machine learning model.

For this tutorial, we want to create a machine model that will allow botanists to classify different species of iris flowers.

The way our model will predict the types of flowers is by learning from their features. These features are the length and width of the petals and the sepals.

By learning from these measurements, our model will predict whether the flower is a setosa, versicolor, or virginica.

Since we are training the model using labeled data for which we already know the answer, this is a supervised machine learning problem.

For instance, when we feed our model with unclassified iris species, it will give us predictions based on its learning from training data.

In other words, this problem is also a classification problem as it involves grouping data into different classes.

For problems such as this, we address possible outputs (different iris species) as classes.

Now open your Jupyter notebook and type in the following imports:

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib

Part 1 - Data Preparation

In this part, we will prepare and analyze the data. Our data is the iris dataset. An advantage of using Scikit-Learn is that it already includes the iris dataset for us.

We just have to load it by calling the load_iris. So type the following:

from sklearn.datasets import load_iris
iris_dataset = load_iris()
print("Keys of iris_dataset:\n", iris_dataset.keys())

Output:

Keys of iris_dataset:
 dict_keys(['data', 'target', 'target_names', 'DESCR', 'feature_names', 'filename'])

The output that we get is a Bunch object. This type is similar to a dictionary as it contains keys and values. Therefore, we can see key-value pairs.

First, the data key is a NumPy array that contains measurements of sepal length, sepal width, petal length, and petal width for 150 different flowers.

Type the following to learn more:

print("Data Type:", type(iris_dataset['data']))

Output:

Data Type: <class 'numpy.ndarray'>

Each row in this NumPy array corresponds to a flower. On the other hand, the columns correspond to their measurements.

Let’s take a look at the shape of the array to understand our data better.

print("Shape of Data:", iris_dataset['data'].shape)

Output:

Shape of Data: (150, 4)

The shape confirms that our data has 150 rows. And each row has 4 columns.

So, when it comes to machine learning, the items or rows are known as samples. And the columns are features.

We can take a look into the features of the first ten samples from our data set. Type the following:

print("First 10 Samples and Their Features:\n", iris_dataset['data'][:10])

Output:

First 10 Samples and Their Features:
 [[5.1 3.5 1.4 0.2]
 [4.9 3.  1.4 0.2]
 [4.7 3.2 1.3 0.2]
 [4.6 3.1 1.5 0.2]
 [5.  3.6 1.4 0.2]
 [5.4 3.9 1.7 0.4]
 [4.6 3.4 1.4 0.3]
 [5.  3.4 1.5 0.2]
 [4.4 2.9 1.4 0.2]
 [4.9 3.1 1.5 0.1]]

Now moving onto the next key, which is the target.

The target key is a NumPy array that holds the iris species encoded as integers from 0 to 2.

And most importantly, this key is a one-dimensional array. We can find out more about the target through the following lines of code:

print("Type of Target:", type(iris_dataset['target']))
print("Shape of Target:", iris_dataset['target'].shape)
print(iris_dataset['target'])

Output:

Type of Target: <class 'numpy.ndarray'>
Shape of Target: (150,)
[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2
 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
 2 2]

As a result, the outputs tell us that this is a one-dimensional NumPy array. Moreover, it has 150 species encoded as integers from 0 to 2.

0 means setosa, 1 means versicolor, and 2 means virginica.

Next, we have the target_names. This key holds an array of strings. These strings are the name of the species of the flowers.

Let’s take a look:

print("Target names:", iris_dataset['target_names'])

Output:

Target names: ['setosa' 'versicolor' 'virginica']

Next, we have the key DESCR. This key provides us with a description of the iris dataset.

Let’s take a look:

print(iris_dataset['DESCR'])

Output:

.. _iris_dataset:

Iris plants dataset
--------------------

**Data Set Characteristics:**

    :Number of Instances: 150 (50 in each of three classes)
    :Number of Attributes: 4 numeric, predictive attributes and the class
    :Attribute Information:
        - sepal length in cm
        - sepal width in cm
        - petal length in cm
        - petal width in cm
        - class:
                - Iris-Setosa
                - Iris-Versicolour
                - Iris-Virginica

    :Summary Statistics:

    ============== ==== ==== ======= ===== ====================
                    Min  Max   Mean    SD   Class Correlation
    ============== ==== ==== ======= ===== ====================
    sepal length:   4.3  7.9   5.84   0.83    0.7826
    sepal width:    2.0  4.4   3.05   0.43   -0.4194
    petal length:   1.0  6.9   3.76   1.76    0.9490  (high!)
    petal width:    0.1  2.5   1.20   0.76    0.9565  (high!)
    ============== ==== ==== ======= ===== ====================

    :Missing Attribute Values: None
    :Class Distribution: 33.3% for each of 3 classes.
    :Creator: R.A. Fisher
    :Donor: Michael Marshall (MARSHALL%PLU@io.arc.nasa.gov)
    :Date: July, 1988

The famous Iris database, first used by Sir R.A. Fisher. The dataset is taken
from Fisher's paper. Note that it's the same as in R, but not as in the UCI
Machine Learning Repository, which has two wrong data points.

This is perhaps the best known database to be found in the
pattern recognition literature.  Fisher's paper is a classic in the field and
is referenced frequently to this day.  (See Duda & Hart, for example.)  The
data set contains 3 classes of 50 instances each, where each class refers to a
type of iris plant.  One class is linearly separable from the other 2; the
latter are NOT linearly separable from each other.

.. topic:: References

   - Fisher, R.A. "The use of multiple measurements in taxonomic problems"
     Annual Eugenics, 7, Part II, 179-188 (1936); also in "Contributions to
     Mathematical Statistics" (John Wiley, NY, 1950).
   - Duda, R.O., & Hart, P.E. (1973) Pattern Classification and Scene Analysis.
     (Q327.D83) John Wiley & Sons.  ISBN 0-471-22361-1.  See page 218.
   - Dasarathy, B.V. (1980) "Nosing Around the Neighborhood: A New System
     Structure and Classification Rule for Recognition in Partially Exposed
     Environments".  IEEE Transactions on Pattern Analysis and Machine
     Intelligence, Vol. PAMI-2, No. 1, 67-71.
   - Gates, G.W. (1972) "The Reduced Nearest Neighbor Rule".  IEEE Transactions
     on Information Theory, May 1972, 431-433.
   - See also: 1988 MLC Proceedings, 54-64.  Cheeseman et al"s AUTOCLASS II
     conceptual clustering system finds 3 classes in the data.
   - Many, many more ...

Moreover, you can also print a specific part of the DESCR instead of the complete description.

print(iris_dataset['DESCR'][:500] + "\n...")

Output:

.. _iris_dataset:

Iris plants dataset
--------------------

**Data Set Characteristics:**

    :Number of Instances: 150 (50 in each of three classes)
    :Number of Attributes: 4 numeric, predictive attributes and the class
    :Attribute Information:
        - sepal length in cm
        - sepal width in cm
        - petal length in cm
        - petal width in cm
        - class:
                - Iris-Setosa
                - Iris-Versicolour
                - Iris-Virginica

The key feature_names is a list of strings that provides us with the names of available features.

Go ahead and type the following:

print("Feature Names:", iris_dataset['feature_names'])

Output:

Feature Names: ['sepal length (cm)', 'sepal width (cm)', 'petal length (cm)', 'petal width (cm)']

As you can see we get a list of all the features and their unit.

For now, these are the crucial keys that should allow us to get familiar with the data. You can go ahead and learn more about them by performing various operations with the dataset. But we are good to move onto the training and testing phase.

Part 2 - Training and Testing Data

In this part, we will split our data into two sets, the training set and the testing set.

Training set is the dataset which we use to train our model. The KNN algorithm will learn from the training first how to distinguish the different iris species.

During the training phase, we don’t expose our testing dataset to the model. As a result, when the training is over, we use the testing dataset to measure its success.

Testing dataset is what we use to assess how well our model works.

One of the benefits of using Scikit-Learn is that it comes prebuilt train_test_split() function. This function will split 75% of the rows in the dataset into a training set. And the rest 25% into the testing set.

So type the following code and then I will explain what’s going on.

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(
    iris_dataset['data'], iris_dataset['target'], random_state=0)

The first line of code imports the train_test_split() function from the module sklearn.model_selection.

Next, we are using the train_test_split() function to randomize and split our dataset into the following variables:

X - These are the inputs or features that you feed into your model. This value corresponds to a two-dimensional array (matrix). As a result, we capitalize the X.
y - These are the expected outcomes or the labels. It corresponds to a one-dimensional array (vector).
X_train - This is the training dataset.
y_train - The labels that to the training dataset (X_train).
X_test - This is the testing dataset.
y-test - The labels that to the training dataset (X_test).

And the parameters that we passed are our data itself and the target. I like to think of this function as train_test_split(X, y), where X corresponds to features and y to the labels.

If you can remember, our data holds a two-dimensional array of features to each species. And the target contains a one-dimensional array of labels encoded as integers from 0 to 2.

One more parameter that we are using is the random_state. And we set it to 0. Therefore, it ensures that our function always splits the same data every time we run the train_test_split() function.

So if you don’t assign a fixed value like 0, 1, or 42, then every time you run the code, our training and testing datasets will have different sets of values.

Let’s see what we have so far by printing out their shapes. First, the training dataset:

print("X_train Shape:", X_train.shape)
print("y_train Shape:", y_train.shape)

Output:

X_train Shape: (112, 4)
y_train Shape: (112,)

Now the test dataset:

print("X_test Shape:", X_test.shape)
print("y_test Shape:", y_test.shape)

Output:

X_test Shape: (38, 4)
y_test Shape: (38,)

We see that X_train contains 75% of all the rows. On the other hand, X_test holds 25%.

Part 3 - Data Visualiztion

It is always a good idea to visualize your data for further inspection before creating a model.

At this point, we will visualize our data to see what we have. And one of the best ways to visualize our data is through scatter plots.

Scatter plots are diagrams where we represent our data using dots. We put one feature on the x-axis and one on the y-axis.

One of the purposes of a scatter plot is to observe the relationship between variables.

To create a scatter plot, first, we will create a dataframe from the data X_train using Pandas. After that, label the columns with the strings we saw in feature_names.

Type the following:

df = pd.DataFrame(X_train, columns=iris_dataset.feature_names)
df.head()

Now it is time to convert the dataframe into a scatter plot. We will use the function plotting.scatter_matrix() to pass our dataframe and color our data points based on the labels from y_train.

Type the following:

pd.plotting.scatter_matrix(df,c=y_train,figsize=(12,12), marker='o',s=20,alpha=.8)
plt.show()

From the code, you can see that we have passed several other parameters to modify our chart as we like. For example, figsize lets us determine the size of the diagrams or figures.

You can learn more about the parameters that are available to the plotting.scatter_matrix() function through the official Pandas documentation.

Here’s a challenge for you. Go ahead and try to visualize the X_test data using the same procedure.

See if you can figure it out. You can use the code snippet above as an example.

However, if not, then don’t worry about it. Here’s how you do it:

df = pd.DataFrame(X_test, columns=iris_dataset.feature_names)
pd.plotting.scatter_matrix(df,c=y_test,figsize=(12,12), marker='o',s=20,alpha=.8)
plt.show()

Part 4 - Creating the Model

Building the model using Scikit-Learn is not as complicated as it sounds. As you may already know that it comes with many classification algorithms. So the one that we will import is the KNN algorithm or K-nearest neighbors classifier.

First, we have to import the KNeighborsClassifier class from the neighbors module. After that, instantiate an object of that class.

Then we will set the value of K to 1 using the parameter n_neighbors=1.

And here’s the code for all of that:

from sklearn.neighbors import KNeighborsClassifier
knnObject = KNeighborsClassifier(n_neighbors=1)

The next step is to fit our training data (X_train, y_train) with the knnObject using the method fit().

Think of fitting the data as training. The fit() method allows us to take in our training dataset and train them to make predictions.

So, type the following:

knnObject.fit(X_train, y_train)

Output:

KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',
                     metric_params=None, n_jobs=None, n_neighbors=1, p=2,
                     weights='uniform')

For now, you don’t have to worry about the parameters we got as an output. All we did was fit the model using our training dataset so that we can make predictions.

Part 4 - Making Predictions

Imagine you are a machine learning engineer for a company. A client of yours reached out to you to verify an iris species they found in the wild.

They only gave us the following information:

Sepal length: 40 cm
Sepal width: 10 cm
Petal length: 5 cm
Petal width: 2 cm

Based on these features, we have to make a prediction.

So, our first step is to create a two-dimensional NumPy array and calculate the shape:

newIris = np.array([[40, 10, 5, 2]])
print("newIris Shape:", newIris.shape)

Output:

newIris Shape: (1, 4)

The reason we have created a two-dimensional array is that Scikit-Learn only expects two-dimensional arrays.

Now we are going to make the prediction by calling the predict()method on knnObject:

prediction = knnObject.predict(newIris)
print("Prediction Value:", prediction)
print("Predicted Target Name:",
       iris_dataset['target_names'][prediction])

Output:

Prediction Value: [2]
Predicted Target Name: ['virginica']

The iris our client found belongs to class 2, which corresponds to the species virginica.

But before we confirm our result to the client, we have to make sure that our model predicted the correct species. And that’s where model evaluation comes into play.

Part 5 - Model Evaluation

For model evaluation, we have to use the test set. Even though you don’t use that data to build the model, but only to test it. (If you wonder why the test set stays untouched, overfitting vs underfitting explains it.)

To clarify, this process involves predicting each iris on the test dataset using X_test. And then compare that prediction against the test data’s labels or the y_test.

First, we will make the prediction using knnObect.predict():

testSetPredictions = knnObject.predict(X_test)
print("Test Set Predictions:", testSetPredictions)

Output:

Test Set Predictions: [2 1 0 2 0 2 0 1 1 1 2 1 1 1 1 0 1 1 0 0 2 1 0 0 2 0 0 1 1 0 2 1 0 2 2 1 0
 2]

Then call the score method to measure the model’s accuracy:

accuaracy = round(knnObject.score(X_test, y_test),2)
print("The Test Set Accuracy is:",accuaracy)

Output:

The Test Set Accuracy is: 0.97

In short, our model made the correct prediction of the iris species in the test dataset 97% of the time.

There are ways to tune our model to improve its accuracy and performance. But we will not go into that right now.

To sum up, 97% accuracy is a trustworthy model for such scenarios. However, depending on requirements, that may not be enough. And that’s where model tuning comes into play.

Conclusion

In conclusion, we have created a simple iris classification model using the KNN algorithm. Not only that, but you have also learned how the KNN algorithm works and how to implement it using Scikit-Learn.

Additionally, some of the aspects that we discussed require prior knowledge of specific tools and concepts. So you may need to brush up on those skills.

Besides, focus on learning how to read the official Scikit-Learn documentation. That is a great help.

Found this tutorial useful? Still, having problems understanding the KNN algorithm? What other algorithm do you think we can use to classify iris species?

Pandas Tutorial for Beginners: The Ultimate Guide

Sat, 06 Mar 2021 00:00:00 GMT

If you are an aspiring data scientist or interested in data analysis, you must know working with Pandas. So in this Python Pandas tutorial, I will break down the basics and show you how to work with Series and DataFrame.

Moreover, I will show you how to use Pandas to read, clean, transform and store data.

What is Pandas?

Pandas is a scientific computing library for data analysis in Python. It was developed in 2008 by Wes McKinney. Moreover, it’s completely open-source and built on top of Numpy.

Some of the key features of Pandas includes:

Provides fast performance to process large data sets.
Able to load data from a wide variety of sources.
Offers data structures and operations to manipulate large data sets.

We will learn about other important features as we go along.

Installing Pandas

Before we continue with our Pandas tutorial, I am assuming that you have basic Python knowledge.

To install Pandas, I recommend that you download the Anaconda distribution of Python.

This distribution is an open-source data science platform that comes with Pandas & other scientific libraries.

If you do not want to use the Anaconda, then you can also install Pandas from your terminal using the commands below in the following order:

pip install numpy
pip install pandas

Other than that, Google has a free Jupyter notebook platform known as Google Colab that already provides Pandas and other Python data science libraries. You can use that as well.

Visit here to learn more about Google Colab:

https://colab.research.google.com/

Series

In this part of our Pandas tutorial, I will talk about Series.

Pandas Series is a one-dimensional labeled array to hold any data type such as integer, float, string, and Python objects.

In short, you can think of Series as columns as in excel sheet. A series represents a single column in the computer memory.

Let’s see how we can create a Series.

First, we have to import Pandas using the import keyword.

import pandas as pd

Now, type following to create a Series:

mySeries = pd.Series([0,1,2,3,4,5,10])
print(mySeries)

Output:

0     0
1     1
2     2
3     3
4     4
5     5
6    10
dtype: int64

Also, you can give your Series a name.

mySeries = pd.Series([0,1,2,3,4,5,10], name='Num')
print(mySeries)

Output:

0     0
1     1
2     2
3     3
4     4
5     5
6    10
Name: Num, dtype: int64

Since Pandas works on top of Numpy, the Series contains a Numpy array within itself.

And we can extract the Numpy array using the method values.

mySeries.values

Output:

array([ 0,  1,  2,  3,  4,  5, 10])

As you can see, that the output is a Numpy array.

A Series also uses index positions.

mySeries.index

Output:

RangeIndex(start=0, stop=7, step=1)

To extract values at certain index positions, I can type:

print(mySeries[6])
print(mySeries[4])

The values at index position 6 & 4:

10
4

Similar to other data structures, Python also allows us to slice a Series. For instance, I want to extract the values between the index positions 5 & 7. For that, we can type:

mySeries[5:7]

Output:

5     5
6    10
Name: Num, dtype: int64

Remember that the column on the left holds the index values or positions.

We can also assign names to the indexes when we create a Series.

So let’s create a new Series with student scores. Besides, I will also pass the index parameter.

scores = pd.Series([70,90,80,100,95,85], index=['Sam','Andrea','Marcos','Peng','Karen','Chen'])
print(scores)

Output:

Sam        70
Andrea     90
Marcos     80
Peng      100
Karen      95
Chen       85
dtype: int64

I passed an index parameter, and this parameter is a list object. Furthermore, this list holds all of our index names or labels.

The left column is the index column or, you can refer to it as the label column. On the right, we have all our values.

If you wish to extract all the index labels, then type:

scores.index

Output:

Index(['Sam', 'Andrea', 'Marcos', 'Peng', 'Karen', 'Chen'], dtype='object')

According to the dtype, we have a collection of Python objects.

Indexing & Slicing Series

I did touch a little bit on indexing & slicing Series objects previously. However, let’s try out some more examples to see how they really work.

Let’s perform some indexing and slicing operations using numbers on the Series scores.

For example, I want to check the score of Karen at the index position 4. So, I can type:

scores[4]

Output:

Furthermore, we can slice the Series using numbers as well. Let’s say I want the scores of Andrea, Marcos, Peng & Karen.

Therefore, I have to slice from the index position 1 to 5

scores[1:5]

Output:

Andrea     90
Marcos     80
Peng      100
Karen      95
dtype: int64

When we perform slicing, the ending number is the number up to which Python slices. But it does not include the value.

For example, here, we are slicing up to the index position 5. But not including the actual value.

You can also slice a Series using the index labels.

For example:

scores['Sam':'Peng']

Output:

Sam        70
Andrea     90
Marcos     80
Peng      100
dtype: int64

Notice how slicing with index labels is different compared to the way we slice using numbers.

In contrast to using numerical indexes, you can see that the value of the ending index label (Peng) is in our result.

iloc & loc

Pandas have a built-in object known as the iloc to extract data using the integer-based index.

So, let’s see how we can use the iloc to extract information from scores:

scores.iloc[0:2]

Output:

Sam       70
Andrea    90
dtype: int64

Again, the iloc tells Python to extract data using the integer-based index.

Similarly, we also have the loc object at our disposal to extract data using a label-based index.

This is how it works:

scores.loc['Sam':'Peng']

Output:

Sam        70
Andrea     90
Marcos     80
Peng      100
dtype: int64

You may learn more about iloc & loc when we get to the DataFrame section. But for now, let’s keep it simple.

Before I go to the DataFrame section of this Pandas tutorial, I like to show you two more ways to create Series.

You can also create Series from a Numpy array. First, import Numpy and Pandas into your IDE:

import numpy as np
import pandas as pd

Then create a Numpy array:

data = np.array(['X','Y','Z'])

To convert data into a Pandas Series, I can type:

mySeries = pd.Series(data)
print(mySeries)

Output:

0    X
1    Y
2    Z
dtype: object

Series from Dictionaries

We can also create Pandas Series from Python dictionaries.

So let’s start by creating a dictionary:

dictScores = {'Jacob':98,'Mae':70,'Kayla':95}

Now pass the dictScores into pd.Series() method:

mySeries = pd.Series(dictScores)
print(mySeries)

Output:

Jacob    98
Mae      70
Kayla    95
dtype: int64

When creating a Series with a dictionary, you should remember that the dictionary keys are index labels.

Well, that’s it for Series. In the next part of the Pandas tutorial, we will talk about DataFrames.

DataFrames

Great job if you have made it through Series. In this part of the Pandas tutorial, we learn about DataFrames and the multiple ways we can create them. Not only that, but you will also learn how to set indexes and select, combine and create columns on a DataFrame.

Now, what is a DataFrame in Pandas?

A DataFrame is a two-dimensional data structure that has rows and columns. These columns and rows are known as labeled axes.

And columns of a Dataframe are made up of separate or multiple Series objects. In short, one of the ways you can create a DataFrame is by having two or more Series.

Coming back to the axes, a DataFrame has two axes. They are axis 0 & axis 1.

In simple terms, axis 0 represents rows and, axis 1 represents columns. Sometimes I like to think of Pandas DataFrame as Excel sheets.

Anyway, let’s get our hands dirty with some coding.

First, I will show you how to create DataFrames using a dictionary.

Go ahead and import the following libraries:

import numpy as np
import pandas as pd

Again, create a dictionary objects called scores:

scores = {
  'Name':['Jake','Chan','Alex'],
  'Age': [15,16,17],
  'Grade': ['C','B','A']
}

The keys of our dictionary will be the column names of our DataFrame. And the values of the dictionary will be the list of items under those columns.

Now convert score into a DataFrame:

df = pd.DataFrame(scores)
print(df)

Output:

   Name  Age Grade
0  Jake   15     C
1  Chan   16     B
2  Alex   17     A

As you see, we got ourselves a nice-looking DataFrame.

And this DataFrame looks similar to an excel sheet.

Also, notice that the keys of our dictionary became the column names. Then the values are the list of items under the columns.

On the far left side of the DataFrame, we have the index column. Each index value represents a row.

If you want to check the column names, then you can type:

df.columns

Output:

Index(['Name', 'Age', 'Grade'], dtype='object')

Remember that each column is a Python object it self.

By the way, each row is an index on our DataFrame. So, to see all the index or rows, I can type:

df.index

Output:

RangeIndex(start=0, stop=3, step=1)

The output shows that our index starts at 0 and stops at 3 with a default step of 1.

DataFrames from Series

So far, we have learned how to create Pandas DataFrame using Python dictionary.

Now, I will show you how to create DataFrames using Series.

Step 1 - Create Dictionaries

We will start by creating two dictionaries, with each having yearly scores of students.

So the first one holds student score information from the year 2020. Then the second tells us their score from 2021.

scores2020 = {'Jake':90,'Kayla':85,'Muhammad':95,'Alexis':98}
scores2021 = {'Jake':85,'Kayla':95,'Muhammad':90,'Alexis':97}

Step 2 - Convert the Dictionaries into Series

The keys in our dictionaries are the index labels. So, we have created two dictionaries. Now the next step is to convert them into Series objects using pd.Series().

seriesOne = pd.Series(scores2020)
seriesTwo = pd.Series(scores2021)

Step 3 - Create a New Dictionary Using the Series Objects

In this step, we will create a new dictionary that we will pass into the method pd.DataFrame() later.

Let’s create a newDict first:

newDict = {'2020': seriesOne,'2021': seriesTwo}

The keys of the newDict are the column names. And the values (Series objects) are the data each column holds.

Step 4 - Convert the New Dictionary into a DataFrame

Lastly, we will create the DataFrame by converting the newDict.

For this to work, I have to use pd.DataFrame() and pass our newDict

Here’s what I mean:

yearlyScores = pd.DataFrame(newDict)
print(yearlyScores)

Output:

          2020  2021
Jake        90    85
Kayla       85    95
Muhammad    95    90
Alexis      98    97

Take a look at what’s going on here. I used the pd.DataFrame() method and passed in our newDict where the keys were the columns. And the rows were the values.

Working with Columns

As a data scientist, you will work with Pandas a lot. As a result, you also need to work with columns to make changes or retrieve specific information.

So, let me show you the ways you can work with the columns of a DataFrame.

We will use the yearlyScores DataFrame that we just created.

To select a column, type this:

yearlyScores['2020']

Output:

Jake        90
Kayla       85
Muhammad    95
Alexis      98
Name: 2020, dtype: int64

Similarly, we can also select the year 2021 using the same process:

yearlyScores['2021']

Output:

Jake        85
Kayla       95
Muhammad    90
Alexis      97
Name: 2021, dtype: int64

You can also see that we have the index column on the left. So, no matter what column you select, the output will always have an index column as a default.

It’s time to see how we can create a new column and add it to an existing DataFrame.

I will first create a new Pandas Series, which is technically a column, and then add it to the DataFrame yearlyScores.

Creating a Series from a dictionary:

scores2019 = scores2019 = {'Jake':78,'Kayla':90,'Muhammad':96,'Alexis':100}
seriesThree = pd.Series(scores2019)
seriesThree

Output:

Jake         78
Kayla        90
Muhammad     96
Alexis      100
dtype: int64

After that, I can add this Series to our DataFrame as a new column:

yearlyScores['2019'] = seriesThree
print(yearlyScores)

So we have our usual DataFrame and, then I have used the square brackets to name our new column '2019'. Then I have assigned the yearlyScores['2019'] to the seriesThree object. As a result, it adds a new column to the DataFrame.

Output:

          2020  2021  2019
Jake        90    85    78
Kayla       85    95    90
Muhammad    95    90    96
Alexis      98    97   100

But before we move further, we have a problem with our DataFrame. And that is, we want column 2019 to be at the first instead of last.

We have to reorder it. And to perform reordering operation, I can type:

yearlyScores = yearlyScores[['2019','2020','2021']]

Let’s print it out:

print(yearlyScores)

Output:

          2019  2020  2021
Jake        78    90    85
Kayla       90    85    95
Muhammad    96    95    90
Alexis     100    98    97

The order of the columns is now changed.

One of the aspects of Pandas is that when we create DataFrames from Series, it will automatically match the index and add the column.

You can also delete columns. For instance, I want to delete the column 2019 from our DataFrame. And to do that, I can use the del keyword:

del yearlyScores['2019']

Print out yearlyScores:

print(yearlyScores)

Output:

          2020  2021
Jake        90    85
Kayla       85    95
Muhammad    95    90
Alexis      98    97

You can see that column 2019 is now gone.

Often you may also need to delete multiple columns. Therefore, we can do that using the drop() method.

Here’s how it works:

yearlyScores.drop(['2020','2021'], axis=1,inplace=True)

So, I want to delete the columns 2020 & 2021. That’s why I passed them as a list. Since we are dealing with columns, I also set axis=1. Lastly, I have inplace=True, which means that we want to make the change permanent. If it is False, then the drop() method would return a copy instead of making the change permanent.

Also, do know that, by default, inplace is always False.

Let’s print it out:

print(yearlyScores)

Output:

Empty DataFrame
Columns: []
Index: [Jake, Kayla, Muhammad, Alexis]

All of our columns are now gone. Python tells us that it is an empty DataFrame.

Similar to working with columns, you can also select, add and delete rows.

Since we dropped all of our columns from the DataFrame, we have to recreate yearlyScores again.

Use the following code to do that:

scores2020 = {'Jake':90,'Kayla':85,'Muhammad':95,'Alexis':98}
scores2021 = {'Jake':85,'Kayla':95,'Muhammad':90,'Alexis':97}

seriesOne = pd.Series(scores2020)
seriesTwo = pd.Series(scores2021)

newDict = {'2020': seriesOne,'2021': seriesTwo}

yearlyScores = pd.DataFrame(newDict)
print(yearlyScores)

Output:

          2020  2021
Jake        90    85
Kayla       85    95
Muhammad    95    90
Alexis      98    97

So, we have our yearlyScores DataFrame back. I kept all the data the same as before for the sake of this tutorial.

Working with Rows

Okay, let’s see how to work with rows.

There are two ways you can select a row. It’s either using the index labels or the index values.

To select rows using labels, I can use the loc function and pass the row name.

For example:

yearlyScores.loc['Muhammad']

Output:

2020    95
2021    90
Name: Muhammad, dtype: int64

The result that we got is a Series. Moreover, notice that the column names of our DataFrame became the index labels.

Then we have the iloc function to select rows. bypassing the integer-based index value.

For instance, I want to extract Kayla’s scores. The index value at which her score exists is 1. As a result, I have to pass the index value 1 to the iloc function.

Here’s how it works:

yearlyScores.iloc[1]

Output:

2020    85
2021    95
Name: Kayla, dtype: int64

Besides, we can also slice multiple rows using the colon operator. For example, I want to extract all the scores starting from Kayla to Alexis.

And the procedure is the same as how you would slice a Python list.

So, I will type:

yearlyScores[1:4]

Output:

          2020  2021
Kayla       85    95
Muhammad    95    90
Alexis      98    97

The output is a DataFrame containing the rows that we sliced starting from the index position one and till but not including the index position four.

Moreover, we can also add new rows to a DataFrame using the append() function.

However, I will create a new DataFrame first. You’ll see why in a bit.

You can create a DataFrame in any way you like. But, I will use Series.

So, this DataFrame will have two new students with their scores.

To create the DataFrame, I can type:

# Create the Series objects
scores2020 = pd.Series({'Lori':80,'Vince':82})
scores2021 = pd.Series({'Lori':90,'Vince':92})

# Pass and convert our Series objects to DataFrame
newStudentsDf = pd.DataFrame({'2020':scores2020,'2021':scores2021})

Output:

       2020  2021
Lori     80    90
Vince    82    92

In the end, we have our new DataFrame. So, let’s add rows using newStudentDf to our previous DataFrame yearlyScores.

Type the following:

yearlyScores = yearlyScores.append(newStudentsDf)
print(yearlyScores)

Output:

          2020  2021
Jake        90    85
Kayla       85    95
Muhammad    95    90
Alexis      98    97
Lori        80    90
Vince       82    92

To sum up, first, I created a new DataFrame, newStudentDf. This DataFrame holds two new students and their scores. Then, I passed that DataFrame using the append() function to merge its rows with yearlyScores.

You may need to go over the whole procedure few times to wrap your head around it. In contrast to columns, rows may be complex.

We are almost at the end of this Pandas tutorial. If you have been able to follow along till this point, then well done! We are almost there.

The last concept that I want to talk about is deleting rows.

Deleting rows is similar to deleting columns. Like before, we can use the drop() function to delete rows in Pandas.

For instance, I want to delete Jake and Lori’s information from our DataFrame. As simple as it sounds, I will type:

yearlyScores.drop(['Jake','Lori'],inplace=True)
print(yearlyScores)

We have to use the inplace parameter to make our change permanent.

So, here is what we have:

          2020  2021
Kayla       85    95
Muhammad    95    90
Alexis      98    97
Vince       82    92

Conclusion

Finally, we are at the end of our Pandas tutorial. I hope you learned some of the valuable concepts on how to work with Pandas.

I recommend going over the tutorial a few times to understand the concepts vividly. Although, it may take a while. To this day, I still have to practice Pandas regularly to keep up my skills sharp.

Working with DataFrame can be complex. That’s why follow the order that I have mentioned when it comes to learning Pandas. Besides, the best way to get good at a skill is by practicing.

How do you plan to use Pandas? Is there any part of this Pandas tutorial that was confusing? What other Python libraries can you think of that are great for data analysis?

Reading and Writing Files in Python

Wed, 29 Jul 2020 00:00:00 GMT

A program can be short or long. It can have several functionalities in it, such as asking input from a user and then giving an output. Eventually, when the program terminates, you will lose your results if you don’t save them. That’s why reading and writing files in Python comes in handy.

Similarly, we may also need to read data or information in our code from an external source. Let’s say you want your program to read data from a text file. That’s why the data needs to be in a format that you can use your application to read later.

In programming, we use files to store data. And there are two types of files:

1. Text files: The text files are simple files used for storing standard text. A text file has a .txt extension, and which you can create easily using a notepad.

2. Bin files: The bin files store data in binary format. A bin file has a .bin extension.

We can use such files our programs using file operations that Python provides. So in this article, we will discuss how to perform file operations and learn the nuts and bolts of reading and writing files in Python.

File Operations in Python

There are two types of file operations when it comes to reading and writing files in Python - Read and Write.

1. Read operation: The read operation reads a file. Suppose we have a text file, and we want to use the text in our program. Then, we have to perform the read operation.

2. Write operation: The write operation writes on a file. When a program terminates, every data associated with it is lost. If we want to store the data, we can use the write operation to save it in a new file, or to an existing one.

Access modes

The access modes determine why a file is open and what kind of operation we can perform. These modes also define their location.

One of the key concepts that you have to understand about reading and writing files in Python are the access modes.

Python has six access modes for accessing files. Let’s discuss each of them with the help of examples.

Read-only

The read-only mode reads a file, and it is denoted by “r.”

In simple words, this mode opens an existing file. Python has an in-built function called open(), which we can use to open our file. Python then places a stream at the beginning of the text to start reading from the very first character.

Think of a stream as a general pointer that points to a particular location on our file.

Let’s see how to use the open() function to read a file in Python.

file1 = open("sample1.txt", "r")
print(file1)
file1.close()

The open() function takes two parameters - location of the file and the access mode.

In the end, we have a close() function. It will deallocate the memory assigned to open the file. Also, reset the access mode.

If the file is present in the same directory in which the program exists, then we can also write the file, else we have to specify the full path.

Remember, r denotes the read-only mode.

The file1 is the object for sample1.txt.

To read the data, Python provides three functions.

1. read()

The read() function reads a file and returns a string.

If there is no argument, then it will read the entire file and give you an output.

This is a sample file.
This is a sample file.
This is a sample file.
This is a sample file.
This is a sample file.
>>>

Moreover, if the argument is a number, the function will read only that many characters.

file1 = open("sample1.txt", "r")
print(file1.read())
file1.close()

The read() function by itself will read the entire file.

Let’s read the first ten characters.

file1 = open("sample1.txt", "r")
print(file1.read(10))
file1.close()

Output:

This is a
>>>

2. readline()

The readline() function is similar to the read() function, but it only reads one line.

file1 = open("sample1.txt", "r")

print("readline: ",file1.readline(40))
print("read: ",file1.read(40))

file1.close()

Output:

readline: This is a sample file.

read: This is a sample file.
This is a sample
>>>

Both read() and readline() functions should read first 40 characters. Let’s see what happens.

While the read() function successfully reads the first 40 characters, the readline() function only reads the first line.

3. readlines()

The readlines() function returns a list of lines in the file.

file1 = open("sample1.txt", "r")
print(file1.readlines())
file1.close()

Output:

['This is a sample file.\n', 'This is a sample file.\n', 'This is a sample file.\n', 'This is a sample file.\n', 'This is a sample file.\n'
>>>

Additionally, the read-only mode is the default mode in Python. As you use the open() function, it will still work without passing the r.

Read & Write

The read and write mode reads and then writes on a file. It is denoted by r+.

The content that we are trying to add or append will be at the end of a file.

file1 = open("sample2.txt", "r+")
print("Before write: ", file1.read())

file1.write("Hello World!")
file1.seek(0)
print("After write: ", file1.read())

file1.close()

Output:

Before write: This is a sample.
After write: This is a sample. Hello World!
>>>

Let’s understand the above code step by step.

First, the code is opening the file using the open() function. We here have the read and write mode, which is r+.

Then, there is the write() function to write on the already opened file. We then used the seek() function and provided 0 as the argument.

file1.seek(0)

0 means the stream will move to the first character.

At last, there is the read() function to read the file.

Write-only

The write-only mode allows us to write on a file. This mode has the denotation w.

file1 = open("sample1.txt", "r")
print("Before: ", file1.read())

file1.close()
file1 = open("sample1.txt", "w")
file1.write("Hello World!")

file1.close()
file1 = open("sample1.txt", "r")

print("After: ", file1.read())

file1.close()

Output:

Before: This is a sample
After: Hello world!
>>>

We cannot use any other function to read the file when the access mode is write-only.

After the write operation, the close() function allows us to open the file again using the read-only mode.

Now, let’s try to read with the write-only mode.

file1 = open("sample1.txt", "w")
file1.write("Hello World!")

print(file1.read())

file1.close()

It will throw an error:

Traceback (most recent call last):
  File "D:\example.py", line 5, in <module>
    print(file1.read())
io.UnsupportedOperation: not readable
>>>

So, when the mode is write-only, we cannot read the file.

Write & Read

Like read and write mode, the write and read mode works to read and write on a file. It has a denotation of w.

But there are a couple of differences. And try to understand them carefully.

The read and write mode throws an error if the file does not exist while write and read will create a new file if it does not exists.

file1 = open("demo.txt", "r+")
file1.write("Hello World!")
file1.seek(0)

print(file1.read())

The above code uses the read and write mode to open a file named demo.txt. You can see that the file does not exist.

Output:

Traceback (most recent call last):
  File "D:\example.py", line 5, in <module>
    file1 = open("demo.txt", "r+")
FileNotFoundError: [Errno 2] No such file or directory: 'demo.txt'
>>>

It throws an error.

Now let’s try the same with write and read mode.

file1 = open("demo.txt", "w+")
file1.write("Hello World!")
file1.seek(0)

print(file1.read())

Output:

Hello World!
>>>

It will create a new file and writes on it.

The write and read mode overwrites the content of the existing file.

After the above example, demo.txt has “Hello World!”.

file1 = open("demo.txt", "w+")
file1.write("Say Hi")
file1.seek(0)

print(file1.read())

The write operation will overwrite “Hello World!” with “Say Hi.”

Append-only

The append-only mode appends characters at the end of the content. It is denoted by a.

file1 = open("sample1.txt", "r")
print("Before: ", file1.read())

file1.close()
file1 = open("sample1.txt", "a")

file1.write("Hello World!")
file1.close()

file1 = open("sample1.txt", "r")
print("After: ", file1.read())

file1.close()

Output:

Before: This is a sample.
After: This is a sample.Hello World!
>>>

While the write-only overwrites the content, the append-only mode appends at the end of a line or a character.

The append mode will also throw an error if we try to read the file.

file1 = open("sample1.txt", "a")
file1.write("Hello World!")

print(file1.read())
file1.close()

Output:

Traceback (most recent call last):
  File "D:\example.py", line 5, in <module>
    print(file1.read())
io.UnsupportedOperation: not readable
>>>

Append & Read

The append and read mode is similar to the append-only but with one exception. It can also read the file.

file1 = open("sample1.txt", "a+")
file1.seek(0)

print("Before:", file1.read())

file1.write("Hello World!")
file1.seek(0)

print("After: ", file1.read())
file1.close()

It will not throw an error:

Before: This is a sample.
After: This is a sample.Hello World!
>>>

Points to Remember

There are few functions and six access modes in total for reading and writing files in Python.

Let’s discuss some of the essential points you should keep in mind while performing read and write operations in Python:

1. Always be careful while specifying the access mode.

2. Always close the file using the close() function. It will deallocate the memory.

3. Use the read() and readline() function carefully. Remember, the readline() function will only read the first line.

4. The read and write mode do not create a new file while the write and read mode does.

5. The write mode will overwrite the existing content of a file. It is similar in the case of write and read mode.

6. The append mode will append at the last of the content. It is similar in the case of append and read mode.

7. Always use the seek() function with 0 as the argument if it is required to move the stream to the beginning.

Conclusion

I recommend that you go over this tutorial again and again until you are clear with the modes. This may take a while.

For structured data, JSON with Python is a natural next step.

Is there any part of this tutorial that is confusing you? Do you like reading and writing files in Python? Are there any mistakes in this article?

JSON with Python: Reading & Writing (With Examples)

Sun, 26 Jul 2020 00:00:00 GMT

Modern software development is very much dependent on storing and exchanging data. The last two decades have seen many ways of storing and transferring data. Today, One of the efficient and accessible ways is using JSON with Python.

JSON stands for Javascript Object Notation. One another popular way is using XML. It isn’t easy to parse, read, and write in XML when compared to JSON. The syntax of JSON comes from Javascript object literals. But JSON is in the text format. You can generate JSON through any programming language and quickly parse it.

As mentioned above, JSON is javascript objects in text format.

Observe the following code:


{

    name : "John",
    age : 25,
    location : "New York"

}

The following is the JSON format of the above data:

{

    "name" : "John",
    "age" : 25,
    "location" : "New York"

}

You can see both are very similar. In the JSON, we write keys inside quotes. JSON files uses the .json extension. You have to store the JSON file with the .json extension.

The most common use of JSON is exchanging data between a REST API client and a web server.

JSON with Python

JSON data is a string. Let’s say you want to convert a Python dictionary. To do that, you have to encode the dictionary as a series of bytes.

This process is called serialization. Similarly, the reverse of serialization is deserialization. It means, decoding JSON into a python object. Let’s discuss serialization and deserialization in Python using the in-built methods.

Python also comes with the JSON module.

To work with JSON in Python, you need to import the JSON module in your python file. You can import the JSON module using the following statement:

import json

Serialization

Serialization means converting a Python data type into JSON. The JSON encoder only supports data types that are native to Python. These are int, str, bool, float, list, dict, and tuple. You can convert the native data types into JSON.

The JSON module has two methods for serialization - dumps() and dump().

dumps()

The dumps() method takes a Python object as a parameter and returns a JSON string. Here’s an example:

import json

dc = { "name" : "John", "age" : 25, "location" : "New York"}
js = json.dumps(dc)
print(js)

In the above code, I created the dictionary dc, and serialized it using dumps() method using JSON module. Let’s see what is the output:

{"name": "John", "age": 25, "location": "New York"}

It returns a JSON object. You may say that a Python dictionary and the returned JSON looks similar.

Let’s try to convert another data type into JSON.

import json

tp = (1,2,3,4,5)
js = json.dumps(tp)
print(js)

This time, instead of dictionary, I am using a tuple.

Output

[1, 2, 3, 4, 5]

Our code converts the tuple into a JSON array.

There are two additional parameters that you can use with the dumps() method - sort_keys and indent.

Setting the sort_keys to True will sort key-value pairs in the result. We get our results in ascending order.

The indent parameter will customize the indentation of our output according to the value we specify.

Both of these parameters make the result easily readable.

import json

dc = { "name" : "John", "age" : 25, "location" : "New York"}
js = json.dumps(dc, sort_keys = True, indent = 3)
print(js)

The sort_keys is set True and the indent is 3. Let’s check out the output:

{
   "age": 25,
   "location": "New York",
   "name": "John"
}

dump()

The dump() method also converts a python object into JSON, but it stores the resultant data in a file. Observe the following code:

import json

dc = { "name" : "John", "age" : 25, "location" : "New York"}
file = open("newfile.json", "w")
json.dump(dc, file, sort_keys = True, indent = 3)
file.close()

The name of the file is newfile.json, in which we will store the data.

You can see that we are passing the name of the file as the second parameter.

Please have a look at the newfile.json file after we ran the code:

{
   "age": 25,
   "location": "New York",
   "name": "John"
}

The converted dictionary is present in our JSON file.

De-serialization

De-serialization is the reverse process of serialization. It converts encoded JSON data into a Python native data type.

The JSON module has two methods for de-serialization - loads() and load().

loads()

The loads() method takes JSON data as the first parameter and returns a python native object. It is the reverse of the method dumps().

import json

js = '{ "name" : "John", "age" : 25, "location" : "New York"}'
dc = json.loads(js)
print(dc)

I created a JSON object js. Then passed it to the loads() method of the JSON module. Note that JSON data is a string.

And here is our output:

{'name': 'John', 'age': 25, 'location': 'New York'}

load()

Now let’s come to the method load().

The load() method is to read JSON from a file and convert it into a native python data type. First, read the file and then pass it to the load() method.

import json

file = open("newfile.json", "r")
data = json.load(file)
print(data)

file.close()

After reading the newfile.json, it is passed to the load() method. The variable data stores the converted value. Printing it gives:

{'age': 25, 'location': 'New York', 'name': 'John'}

Custom Objects

Until now, we discussed JSON with Python using in-built Python methods.

No doubt, these methods are useful, but in a real-time application, you may need to deal with manually created class-based objects. You can also perform serialization and de-serialization on custom objects.

Observe the following python code:

import json

class DemoClass:
    def __init__(self, val1, val2):
        self.val1 = val1
        self.val2 = val2

obj1 = DemoClass("Value1", "Value2")

We have a class, DemoClass. The __init__() method is initializing two attributes val1 and val2.

Then, we create an instance of DemoClass, obj1.

Let’s serialize this object using the dumps() method and see what we get.

import json

class DemoClass:
    def __init__(self, val1, val2):
        self.val1 = val1
        self.val2 = val2

obj1 = DemoClass("Value1", "Value2")
js = json.dumps(obj1)

print(js)

It throws an error:

TypeError: Object of type DemoClass is not JSON serializable

According to this error, you can not serialize an object of a manually created class.

To solve this problem, Python has a __dict__ attribute. Let’s pass obj1.__dict__ to the dumps() method instead.

import json

class DemoClass:
    def __init__(self, val1, val2):
        self.val1 = val1
        self.val2 = val2

obj1 = DemoClass("Value1", "Value2")
js = json.dumps(obj1.__dict__)

print(js)

Let’s see if it works now:

{"val1": "Value1", "val2": "Value2"}

It works perfectly.

Conclusion

In modern software development, JSON plays a crucial part.

Throughout this article, you may have noticed how I stored data in JSON and made conversions.

Python has handy in-built methods for such serializations and de-serializations. That’s one of the reasons why I love to work JSON with Python so much.

Your JSON usually ends up in a file, so reading and writing files in Python is a good next read.

Do you love to handle JSON with Python? What are your thoughts on this article? Do you have any questions?

NumPy Tutorial for Beginners: Arrays

Mon, 11 Nov 2019 00:00:00 GMT

In this tutorial, we will learn about one of the most powerful scientific computing libraries Numpy. We will also learn what are Numpy’s n-dimensional arrays (ndarrays) and how to create, analyze, and manipulate them using Python.

To follow this article, you will need basic knowledge of the Python programming language.

What is Numpy?

Numpy is a scientific computing package used for performing mathematical and logical operations on arrays. And arrays are data structures that store the value of the same data type.

Numpy itself is written in C. However, Numpy extends Python by integrating powerful data structures by allowing us to use multi-dimensional arrays and matrices.

One of the reasons why Numpy is so useful for data analysis, machine learning & AI is that it already comes with powerful multidimensional array objects to work with large data sets.

Installing Numpy

To install Numpy I suggest that you download the Anaconda distribution of Python, which is an open-source data science platform that comes with Numpy & other Python scientific libraries.

If you don’t want to use the Anaconda, then you can also install Numpy from your terminal using the following command:

pip install numpy

Numpy Arrays from List

Before we jump into Numpy arrays, let’s define again what actually is an array. Well, an array is simply a data structure that contains a group of elements that are all of the same data types.

A Numpy array is also a data structure similar to an array but they are the core data structure of the whole Numpy library and comes in two forms. They are vectors and matrices.

Vectors are strictly 1-dimensional arrays. Where matrix or matrices are 2-dimensional arrays that can hold rows and columns.

One of the most common ways to create a Numpy array from a Python list is by using the np.array() function. The example below shows how you can create a 1-dimensional Numpy array.

# Creating an 1-d array from Python list.

import numpy as np

myList = [0,1,2,3,4,5,6,7,8,9]
myArray = np.array(myList)

print(myArray)

Over here we are creating a 1-dimensional Numpy array using the function np.array().

Output:

[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

It may look like a regular Python list, but it’s not. And we can verify that using the type() function*.*

print(type(myArray))

Output:

class 'numpy.ndarray'

After running the previous line, we see that myArray is of the type numpy.ndarray.

Now, let’s create a 2-d Numpy array.

In order to create a 2-d array, we have to create a list of lists. And then similar to before, we will pass that list onto np.array() function to generate our 2-d Numpy array.

# Creating an 2-d array from Python list.

import numpy as np

myList = [[1,2,3],[4,5,6],[7,8,9]]
myArray = np.array(myList)

print(myArray)

So, what we get is a 2-dimensional array with 3 rows and 3 columns.

Output:

[[1, 2, 3]
 [4, 5, 6]
 [7, 8, 9]]

We can also check the type of the array to verify that it is of type numpy.ndarray.

print(type(myArray))

Output:

class 'numpy.ndarray'

Here we see that our 2-d array or matrix is also an object of type numpy.ndarray. And nd refers to an n-dimensional or multi-dimensional array.

Numpy Arrays Using Built-In Functions

Next up on this Numpy tutorial is creating Numpy arrays using built-functions.

Now that we know, how to create Numpy arrays from regular Python lists, I will go ahead and show you how to create them using built-in functions that Numpy provides.

arange()

Numpy has a built-in function called the arange() function. It is similar to the Python built-in range() function.

Let’s say we want to create an array that contains 5 digits from index 0 to 5. We can type:

myList = np.arange(0,5)
print(myList)

Output:

[0, 1, 2, 3, 4]

Numpy created an array of 5 digits starting from index 0 to 5.

Let’s take another example. Say that we want to generate an array starting from 2 to 10.

myList = np.arange(2,10)
print(myList)

Output:

[2, 3, 4, 5, 6, 7, 8, 9]

Notice how that our output on the previous line gave us an array starting from 2 till 10. But the 10 is actually not included. If you want to grab 10 as well then you can type:

myList = np.arange(2,11)
print(myList)

Output:

[ 2, 3, 4, 5, 6, 7, 8, 9, 10]

The arange() function can also take a third argument which specifies the number of steps or spacing between values.

For example, if you want to generate all the even numbers between 0 to 10, pass 2 as the third argument. This will tell Python how many steps to take between the values inside your array.

myArray = np.arange(0,11,2)
print(myArray)

Output:

[ 0,  2,  4,  6,  8, 10]

All you have to add is to add a step size of 2 to get an array of even numbers. By default, the step size in arange() function is set to 1.

zeros()

There is also a function known as zeros() to generate an array of all zeroes. And it looks like this:

# Generating 1-d array of zeros.

myArray = np.zeros(10)
print(myArray)

Output:

[0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]

What we got here is a 1-dimensional array of 10 zeros. On the other hand, we can also create a 2-d array by passing a tuple.

# Generating 2-d array of zeros.

myArray = np.zeros((3,3))
print(myArray)

Output:

[[0., 0., 0.],
 [0., 0., 0.],
 [0., 0., 0.]]

The first number that we passed represents the number of rows and the 2nd number represents the number of columns. In this case, we passed (3,3) to generate an array of 3 rows and 3 columns.

ones()

The same way we can also generate an array of ones.

# Generating 1-d array of ones.

myArray = np.ones(8)
print(myArray)

Output:

[1., 1., 1., 1., 1., 1., 1., 1.]

And for generating a 2-d array of ones:

# Generating 2-d array of ones.

myArray = np.ones((3,3))
print(myArray)

Output:

[[1., 1., 1.],
 [1., 1., 1.],
 [1., 1., 1.]]

linspace()

There is another useful built-in function that comes with Numpy. We call it the linspace().

And what it does is, it will return evenly spaced numbers over a specific interval. Let’s say we want 5 evenly spaced points between 1 to 2. For that I will type:

linArr = np.linspace(1,2,5)
print(linArr)

Output:

[1.  , 1.25, 1.5 , 1.75, 2.  ]

So our output tells us that, our array starts from 1 and stops at 2. I specified the third argument to be 5 to get 5 evenly spaced points between my start and stop.

I can also generate an array of 50 evenly spaced points between 1 to 2.

linArr = np.linspace(1,2,50)
print(linArr)

Output:

[1.        , 1.02040816, 1.04081633, 1.06122449, 1.08163265,
 1.10204082, 1.12244898, 1.14285714, 1.16326531, 1.18367347,
 1.20408163, 1.2244898 , 1.24489796, 1.26530612, 1.28571429,
 1.30612245, 1.32653061, 1.34693878, 1.36734694, 1.3877551 ,
 1.40816327, 1.42857143, 1.44897959, 1.46938776, 1.48979592,
 1.51020408, 1.53061224, 1.55102041, 1.57142857, 1.59183673,
 1.6122449 , 1.63265306, 1.65306122, 1.67346939, 1.69387755,
 1.71428571, 1.73469388, 1.75510204, 1.7755102 , 1.79591837,
 1.81632653, 1.83673469, 1.85714286, 1.87755102, 1.89795918,
 1.91836735, 1.93877551, 1.95918367, 1.97959184, 2.]

Do remember that our output here is a 1-d vector or 1-d array as we have only one set of square brackets.

Now here is an important thing, don’t confuse linspace() with arange(). The linspace() gives you evenly spaced precise data points between two values. On the other hand, arange() takes the third argument that specifies the increments after each value.

The way you can see this is that you can use linspace() to get more precise evenly spaced data points between two numbers where arange() gives you more larger intervals between two numbers.

eye()

We can also create an identity matrix using Numpy. An identity matrix is a 2-dimensional square matrix where the number of rows is equal to the number of columns.

To create an identity matrix, we have to use the function np.eye() and pass just one argument. It looks something like this:

identityMat = np.eye(8)
print(identityMat)

Output:

[[1., 0., 0., 0., 0., 0., 0., 0.],
 [0., 1., 0., 0., 0., 0., 0., 0.],
 [0., 0., 1., 0., 0., 0., 0., 0.],
 [0., 0., 0., 1., 0., 0., 0., 0.],
 [0., 0., 0., 0., 1., 0., 0., 0.],
 [0., 0., 0., 0., 0., 1., 0., 0.],
 [0., 0., 0., 0., 0., 0., 1., 0.],
 [0., 0., 0., 0., 0., 0., 0., 1.]]

The output shows that we have a 2-dimensional matrix with 8 tows and 8 columns. And it only consists of 1s and 0s.

Also, notice how the diagonals are the 1s and the rest are just 0s.

Numpy Arrays of Random Numbers

You can also generate random numbers using Numpy.

random.normal()

To generate a random number from the normal distribution you can type:

randomNum = np.random.normal()
print(randomNum)

Output:

0.4788677086535386

Obviously the result is most likely to be different for you since Numpy generates this number randomly.

We can also specify how many random numbers we want from the normal distribution.

randomNums = np.random.normal(size=5)
print(randomNums)

Output:

[ 1.80517058, -1.00428261,  0.55339115,  1.28505388, -0.76103492]

We have an array of 5 random values from the normal distribution.

random.uniform()

Now, let’s say we want values from the uniform distribution. For that you can type:

randomNum = np.random.uniform()
print(randomNum)

Output:

0.7361103066273562

And the same way we can generate an array of random numbers from the uniform distribution.

randomNums = np.random.uniform(size=5)
print(randomNums)

Output:

[0.72582994, 0.59795509, 0.47816143, 0.49102028, 0.60275951]

random.rand()

Okay, so what if we want to generate a 2-d matrix with two rows and two columns? I can type:

randomMat = np.random.rand(2,2)
print(randomMat)

Output:

[0.72914557 0.28649648]
[0.57465855 0.55108253]

random.randint()

One last thing that I want to talk about this post is how to generate a certain amount of random integers between certain values.

Let’s say I want to generate a Numpy array of four random integers between 1 and 50. I can type:

randomNums = np.random.randint (low = 1, high = 100, size = 4)
print(randomNums)

Output:

[68, 23, 23, 81]

I passed three arguments. The first two arguments specify that I want the array to be between 1 & 100. And the last one specifies that I only want four values.

Vectors to Matrices

So far in this Numpy tutorial, we saw how to create 1-dimensional vectors or arrays and 2-d matrices. But there’s one more thing that I want to talk about is converting a vector into a matrix in Numpy.

reshape()

Let’s start by generating a random 1-d array of 16 integers.

myArray = np.random.rand(16)
print(myArray)

Output:

[0.98603364, 0.47461217, 0.12793579, 0.75083417, 0.08093329,
 0.68316509, 0.95807723, 0.89187899, 0.06593721, 0.72183955,
 0.71391102, 0.35244838, 0.4700743 , 0.71870386, 0.78268923,
 0.73916623]

We only have one set of brackets and thus our output is a 1-d array.

In order to convert this into a 2-d array or a matrix, we have to use the reshape() function. And this is how it goes:

myArray = myArray.reshape(4,4)
print(myArray)

Output:

[[0.98603364, 0.47461217, 0.12793579, 0.75083417],
 [0.08093329, 0.68316509, 0.95807723, 0.89187899],
 [0.06593721, 0.72183955, 0.71391102, 0.35244838],
 [0.4700743 , 0.71870386, 0.78268923, 0.73916623]]

So our matrix has 4 rows and 4 columns. Since we have converted a vector with 16 integers, I passed 4 and 4 as arguments. If you have an array of 25 values then you have to pass 5 and 5 as arguments.

The reason for this is, the reshape() function can only convert to equal numbers of rows and columns. Since the vector myArray consists of 16 values, we can only use the reshape () function to create a 4x4 matrix.

Conclusion

Well, that’s it for this post. Make sure to go over this post again if any of the concepts confuse you.

Pandas is built on top of NumPy, so the Pandas tutorial for beginners is a good next step.

Did you find any part of this Numpy tutorial confusing? If so, which part?