Web Development: Overview & Basic Concepts

2130 words · 10 minute read
Last updated: Oct 22, 2020 · Published: Sep 20, 2020

You want to learn web development? Then let me give you an overview & some pointers on some important basic concepts:

When non-technical people are thinking of the web, they are thinking of web sites that they can access via their browser. That is why we will focus on that first. The most important concept to understand here is “request and response” aka “client and server”. This is how a website works:

Request-Response Sequence Diagram

  1. Web browser sends a request to a URL like “http://www.markusdosch.com/"
  2. Behind “markusdosch.com”, there sits a server (a fancy name for a computer that “serves” stuff). The server receives the request and generates a response. Very often, this response is just some text, such as “Hello World".
  3. The browser receives the response.
  4. The browser knows how to display the response to the user.

In case of simple text, the browser will simply display this text as-is. More often, the text looks like this: “<h1>Hello World</h1>”. This is HTML code. If the server tells the browser that its response has HTML as content type, the browser will “render” the document instead of simply putting out the raw response text. This is how every website works - the browser interpreting the response and rendering it as a nice document. The browser already has some default styles built-in. But most often, the website developer will want to tell the browser how to exactly render the elements, e.g. using which colors, or which element sizes. The usual way to do it is via CSS.

Using a special type of tag, the “anchor” tag (“<a>”), the browser can render links to other URLs. When a users clicks a link, a new request is initiated, the server generates a new response (a different response, as the URL is different now), and a new website is rendered by the browser. That is the whole magic behind basic websites - request and reponse.

In this basic model, we can make a separation between two disciplines:

  1. Web design & frontend development: This discipline is concerned with creating good-looking HTML & CSS layouts, that present a website’s content in a good way.
  2. Backend development: This discipline is concerned with developing the program that runs on the server and that decides what to do when a request is received (mainly: What kind of response to send back).

Static files vs. backend development

In the simplest way, you don’t need much backend development. Instead, you could simply have some static HTML files on your server and respond with these. E.g., when a request “/” is received, you could answer with a file “index.html”, and when the request is “/blub/hello.png”, the server would answer with the file on path “/blub/hello.png”. For simple websites that just need to display some static content, this is actually enough. It is quite easy to configure your server to act as a simple “static file server”. This website, for example, is such a static website. But instead of writing all HTML files by hand, I generate them using the static site generator Hugo.

Backend development comes into play when you want your responses to be dynamic, i.e. to potentially be different for every request. E.g. you might want to include the current time in the response, or save or display some data from a database. And this is were things become really interesting on the backend side, and an area full of possibilities emerges…

You can do backend development in any programming language. The only requirement is that your program follows the HTTP protocol, because that is how browsers send their requests and expect the responses. Either a language has already built-in support for the protocol, or the language-community provides packages that make interacting with the protocol easy.

HTTP

What is the HTTP protocol about? Basically, it specifies a set of methods for client-server communication, with the most important ones being GET and POST. A client, e.g. the web browser, initiates a request to the server and specifies which request it is following. As part of the request, some data is sent to the server. The most important being the URL. Less user-visibile data includes headers (e.g. with cookies), but this is out of scope for this document. A GET request is basically just sending a URL to the server. The URL can include parameters, the so-called “query parameters”. Example: http://markusdosch.com/blub?language=de_DE&name=Tom sends two query-parameters to the server. The server can read these parameters and adjust its response according to the parameters (e.g. greet the user with the name parameter). A POST request can additionally contain a request body. The request body can be of any type - in the simplest form, the body’s content takes the same form as the URL query string, e.g. language=de_DE&name=Tom.

Sending data via the request body has some advantages over sending it over the URL. One is that it allows much more content to be sent (a URL has a certain character limitation). Also, as the body can take any form, you could also send data in other formats, e.g. JSON or XML. This might not seem important now, but becomes important once AJAX comes into play later. When the client sends a body, it has to specify the body’s content type. In the case of the “URL query string equivalent”, the content type is called application/x-www-form-urlencoded.

Forms

Why is it “x-www-form-urlencoded”? Because before the advent of JavaScript as client-side language (more on that later), HTML forms were the main way of accepting user input and sending it to the server. In many ways, they still are.

Here is an example HTML form that a browser would be able to render:

1
2
3
4
5
<form action="http://markusdosch.com/submit"  method="GET">
  <input type="text" name="name" placeholder="Please enter your name" required />
  <input type="number" name="age" min="12" max="99" required />
  <input type="submit" value="submit" />
</form>

What does it do? When a user fills in the form data and clicks “submit”, a GET request will be issued to http://markusdosch.com/submit?name=<NAME_INPUT_VALUE>&age=<AGE_INPUT_VALUE>. If the form had set method="POST", a POST request to http://markusdosch.com/submit with body name=<NAME_INPUT_VALUE>&age=<AGE_INPUT_VALUE> (and content type application/x-www-form-urlencoded). In any case, the server could take this request, read the request parameters, react to them, and return a response. There are certain conventions on what type of effect what HTTP request should have (e.g. that GET should not change or create anything user-visible on the server. So subsequent GET requests always return the same result), but this is out of scope here.

Just knowing about HTML forms & backend programming can take you quite far with developing a website. You could e.g. now implement a rudimentary version of Twitter, Facebook, Wordpress etc. On a conceptual level, you now have all the tools to create such CRUD (create-read-update-delete) applications. Of course, these businesses are infinitely more complex and have much more logic going on, but still, on a conceptual level, they are not that hard.

Client-Side Programming

Requests from browser, some business logic on the backend, resulting in a response that gets rendered by the browser - this could all be quite easy. But has a big downside: For every user interaction, a complete fresh page load has to be done. This limits the scope of applications that can be realized. Some simple patterns that would not be possible: Endless scroll in Facebook, submitting a comment while watching YouTube, telling you a username is taken while you are typing. Some large patterns that would not be possible or look very different from what we are used to today: Applications with high interactivity like Google Docs, multiplayer browser games, chats…

This is why client-side programming is a thing. Over the years, browsers have changed from simply rendering static responses, to allowing a full programming language commands to be executed within the browser context. This is very important - this is actual code & logic being executed on the user’s own machine, instead of on the server! This is huge!

Important: Browsers only execute one programming language: JavaScript. This is changing with WebAssembly (Wasm), but out of scope here.

The web server can send JavaScript code alongside the HTML/CSS code (in special <script> tags, and some other special locations). The browser will automatically execute the code when rendering the page. As part of the code execution, JavaScript can register event listeners, e.g. execute a function when a users clicks on a button, or on an image, or presses a key.

JavaScript makes the difference between “web design” (using HTML and CSS) and actual “frontend development” (or “frontend engineering”).

The two most important things that JavaScript enables are:

  1. AJAX: Sending requests to a server without triggering a page reload (and handling the response).
  2. DOM Manipulation: The DOM is the page’s HTML/CSS structure. JavaScript allows us to modify it after the page load has already occurred.

The effect of JavaScript on web sites is huge. Now, web sites could react to events themselves, query a server in the background, and display the result in the page’s DOM. Everything without a new page load. Completely new interactivity patterns & desktop-like web applications are possible.

Frontend-heavy websites (and API-only backends)

Now, with JavaScript on the client-side, the frontend can do many of the tasks that were previously part of the backend. Most importantly, the generation of the HTML that gets rendered by the browsers, and page transitions. Taking it to the extreme, the server response for “frontend-heavy sites” is 100% JS, and 0% HTML & CSS. All the HTML & CSS code would then be generated by the JavaScript code when the response is fully loaded. Subsequently, when the user wants to transition e.g. from a blog overview page to an individual blog post, the browser would ask the server specifically for the blog post contents, and not the full HTML. Then, the client would render the blog post contents into the existing DOM (=> no new page load needed). The backend degenerates to a pure API (in most of the cases, with JSON as communication format). And the frontend mainly consists of JavaScript. In many of these cases, the frontend can then be served as static HTML, CSS, and JavaScript files.

The leaves us with a clear separation between an app (frontend) part, and an api (backend) part, that communication via an API. This is approach is often taken with high-interactivity desktop-like web apps. The clear separation has some advantages, e.g. it becomes easier to split the work between different teams, and technical choices (e.g. deployment, build pipeline) can be optimized differently. On the other hand, this is not an approach well-suited for every project: A lot of JavaScript means that the user’s browser has to do much more work than with “backend-heavy websites”, where the server would do most of the work. Users need faster machines and their batterys suffer. If done badly, frontend-heavy websites can feel much slower than backend-heavy websites. This is a bit ironic, as speed was originally one of the big promises of frontend-heavy websites. As frontend-heavy websites are a relatively young approach with many moving parts, it is quite easy to get things wrong.

My recommendation would be to think hard which approach you want to go - if your website is mostly about showing content & having some light interactivity like comments and forms, backend-heavy approaches are still the way to go. They have well-established best practices, and scale to most user numbers you can imagine. For added interactivity, you can add some sprinkles of JavaScript.

If your website is actually more of a “desktop-app gone web” (think Word, PowerPoint, Photoshop,…), then you should look into the frontend-heavy direction. You have much more options for interactivity there, but as best practices are still emerging, you likely have to make more choices on your own.

Final Words

This was an overview on what I think of the most important concepts in web development. We have started from the request-response model (aka “client and server”) and have seen the separation into “frontend” and “backend” web development. Then, we have taken a closer look at the HTTP protocol, its different request methods, and how data gets transferred from the client to the server (query parameters, request body), e.g. via forms. Then, we have introduced client-side programming with JavaScript and how it can lead to frontend-heavy websites (as compared to more backend-heavy websites).

I think, a lot of the tools you are reading about in the internet (e.g. in guides, blog posts, or discussions) become much clearer and easier to learn once you have the big picture of web development in mind. I hope to have done my part in showing you the “big picture”.

I hope the web is clearer for you now! 🤓🕸

If you have any questions or ideas for follow-up paragraphs or articles, don’t hesitate to message me (mail@markusdosch.com)!