Custom Domain Name During Development With SSL/HTTPS support

So you want to access your web app during development as https://webapp.local.test? I have a development setup on a Ubuntu where I develop a webapp using both frontend (React/ES6) and backend (Java). At certain point of the development you will start to experience problems in the development. The webapp running in the browser is not behaving correctly: typically there are the common reasons behind it like CORS, CSRF, SameSite policy, SecureOnly cookies, etc. These reasons are tightly coupled to the application which is why you should set up a custom domain name for your development needs – it will solve both frontend and backend problems at once!

Development

  • Frontend: JS React app on localhost port 3000
  • Backend: Java SpringBoot app on localhost port 8000
  • Services: Redis, PostgreSQL, Hashicorp Vault, etc.

Problems

  • Frontend related: CORS, CSRF, SameSite, SecureOnly cookies, etc.
  • SSL/TLS i.e. https
  • Domain name is needed in the application logic

Solution

  • Apache/NGINX proxy. Yes, you should use Web server even for your local development!
  • dnsmasq is a lightweight, easy to configure DNS forwarder, DHCP server software and router advertisement subsystem for small networks. Works on both Linux and Mac! In Ubuntu, the default installed DNS is systemd-resolved, which is used for resolving external (e.g. internet) names
  • locally generated certs

First, you really SHOULD only use “.test” domain (Top-Level Domain): it is reserved for that purpose! And let’s say that your server hostname is going to be “local”. Combine these to get domain name “local.test”.

For SSL we can’t use Letsencrypt services:

The best option: Generate your own certificate, either self-signed or signed by a local root, and trust it in your operating system’s trust store. Then use that certificate in your local web server. See below for details. See letsencrypt.org/docs/certificates-for-localhost/

How to generate your own certificate? There is a good tool for that called mkcert. Also, you shouldn’t use /etc/hosts. After you have made the following configurations, you can access your React app via browser https://fe.local.test and Java backend APIs at https://be.local.test/apis/api (assuming that you are using APIs!) In this setup Apache terminates SSL, but if you need to secure also connection between webserver and appserver you can do it with “stunnel”. Stunnel is a proxy designed to add TLS encryption functionality to existing clients and servers without any changes in the programs’ code. You can secure network traffic between Apache web server and Tomcat application server (and also possibly database/JDBC connection with it) transparently! To summarize the solution:

  1. Local DNS resolution for all “.test” domain to 127.0.0.1. Dnsmasq acts a DNS name server
  2. Use Apache as a proxy server that terminates SSL and sets up multiple virtual hosts e.g. http://www.local.test (Apache default redirect), be.local.test (SpringBoot port 8000), and fe.local.test (Create React App port 3000)
  3. Another reason why adding Apache web server is that you can easily manipulate HTTP security headers which often cause troubles while developing!
  4. SSL is configured with wildcard certificate “*.local.test”. Test accessing “https://fe.local.test”
# DNS resolve to 
sudo apt install -y resolvconf
sudo systemctl status resolvconf.service
sudo systemctl enable resolvconf.service
sudo systemctl start resolvconf.service
echo "nameserver 127.0.0.1" | sudo tee /etc/resolvconf/resolv.conf.d/head

# dnsmasq as a nameserver
sudo apt install -y dnsmasq
# dnsmasq as a name  all *.test requests to localhost (127.0.0.1)
echo "address=/test/127.0.0.1" | sudo tee -a /etc/dnsmasq.conf
echo "expand-hosts" | sudo tee -a /etc/dnsmasq.conf
sudo service dnsmasq restart

# verify 
ping -c 1 local.test

# Apache2
sudo apt install -y apache2
sudo a2enmod proxy proxy_http ssl
sudo nano /etc/apache2/sites-available/local-dev.conf
# see separate config file
sudo systemctl restart apache2

# verify with browser fe.local.test

# SSL
# install certutil tool
sudo apt install -y libnss3-tools
# follow installation instructions: https://kifarunix.com/how-to-create-self-signed-ssl-certificate-with-mkcert-on-ubuntu-18-04/
# create cert
sudo mkcert local.test '*.local.test' localhost 127.0.0.1 ::1
# add the created certs to Apache: see separate conf file!
sudo nano /etc/apache2/sites-available/default-ssl.conf
sudo a2ensite default-ssl.conf
sudo systemctl restart apache2

# local-dev.conf
# remember also to add ServerName www.local.test to "apache2.conf"!

<VirtualHost be.local.test:443>
    ProxyPass / http://127.0.0.1:8000/
    ProxyPassReverse / http://127.0.0.1:8000/
    ServerName be.local.test
    SSLEngine on
    SSLCertificateFile      /home/jari/dev/mkcert/local.test+4.pem
    SSLCertificateKeyFile /home/jari/dev/mkcert/local.test+4-key.pem
    # CORS (restrict access from this origin)
    Header set Access-Control-Allow-Origin https://fe.local.test
    Header set Access-Control-Allow-Credentials true
    Header set Vary Origin
    # SameSite is not supported by Java, so append it to every cookie
    Header edit Set-Cookie ^(.*)$ $1;SameSite=lax
</VirtualHost>
<VirtualHost fe.local.test:443>
    ProxyPass / http://127.0.0.1:3000/
    ProxyPassReverse / http://127.0.0.1:3000/
    ServerName fe.local.test
    SSLEngine on
    SSLCertificateFile      /home/jari/dev/mkcert/local.test+4.pem
    SSLCertificateKeyFile /home/jari/dev/mkcert/local.test+4-key.pem
    # CSP (when defined)
</VirtualHost>
# default-ssl.conf

ServerName www.local.test
ServerAlias *.local.test
SSLCertificateFile /absolute/path/cert.pem
SSLCertificateKeyFile /absolute/path/cert-key.pem

For the development you probably also want to enable CORS as modern browsers have quite strict security by default. For example, SameSite=None setting changed quite recently. CORS you can circumvent by using wildcard operator. Note that headers are put to server!

sudo a2enmod headers
sudo nano /etc/apache2/sites-enabled/local-dev.conf
# Add next line to <VirtualHost be.local.test:443> and restart Apache
Header add Access-Control-Allow-Origin "*"
sudo systemctl restart apache2

P.S. I experienced problems (sudo top) with high CPU after these changes. It turned out that most likely you need both systemd-resolved and dnsmasq running simultaneously, and you can’t easily accomplish the local name resolution with systemd-resolved. I spent hours reading about the issue until I found a workaround: the following is copied from the source (requires restarting both services).

My work-around is to add `DNS=127.1.1.1` into /etc/systemd/resolved.conf. Nothing is listening on 127.1.1.1 so the queries go unanswered. The systemd resolver gets answers only from servers provided to me via DHCP or statically assigned to an interface.

Inside my /etc/dnsmasq.conf I have `server=127.0.0.53`, and resolv.conf has `nameserver 127.0.0.1` which is dnsmasq.

Schema For JSON ala LDAP

Those of the a bit older software professionals know LDAP. Due to a work assignment I had to refresh my mind on it. While working I had the idea that “What if we use LDAP schemas also for JSONs?” I mean there is definitely some common ground, and JSON could benefit from all the work that has been poured to LDAP over time. Also, previously I have written about a new and simplified future JSON format called JKON where arrays are basically deprecated. My idea is quite simple: for a LDAP entry there will a JKON entry. LDAP-style schema means that for an JKON object there are some attributes/fields which are mandatory (LDAP MUST) and optional (LDAP MAY). Also, the simplest schema rule is that an object must have at least one class and unique name defined! IMHO, LDAP-style schema actually fits quite well into JSON-style objects because for each object you defined to what classes it belongs to and values are written as fields / maps straight into the object itself! There are a lot of well-defined LDAP objectClasses and attributeTypes available, and it is quite easy the define your own custom extensions aswell.

# LDAP entry
dn: cn=John Doe,dc=example,dc=com
cn: John Doe
givenName: John
sn: Doe
telephoneNumber: +1 888 555 6789
telephoneNumber: +1 888 555 1232
telephoneNumber: +1 888 555 1299
mail: john@example.com
manager: cn=Barbara Doe,dc=example,dc=com
objectClass: inetOrgPerson
objectClass: organizationalPerson
objectClass: person
objectClass: top

And the corresponding JKON entry

// dn => un (unique name)
// objectClass => class
// multivalued entries => fields of a nested map
{
 "class": {
   "1": "inetOrgPerson",
   "2": "organizationalPerson",
   "3": "person",
   "4": "top"
 },
 "un": "cn=John Doe,dc=example,dc=com",
 "cn": "John Doe",
 "givenName": "John",
 "sn": "Doe",
 "telephoneNumber": {
   "1": "+1 888 555 6789",
   "2": "+1 888 555 1232",
   "3": "+1 888 555 1299"
 },
 "mail": "john@example.com",
 "manager": "cn=Barbara Doe,dc=example,dc=com"
}

Summarized: WebAuthN

The problem with passwords is that they are hard to remember – espcecially if you have many. Remembering all passwords in different services is impossible. Also, passwords are always too short, too long, too everything. It is time to go passwordless with WebAuthN.

  • Demo page showcases different programming languages and technologies using WebAuthN, for example Java. For example, WebAuthN4J is a portable Java library for WebAuthn. For example, WebAuthn4J Spring Security is built on the top of WebAuthn4J. But note that based on documentation it is not clear does it implement “server” part: however Java-WebAuthn-Server implements the Relying Party operations required for a server to support Web Authentication. This includes registering authenticators and authenticating registered authenticators. As mentioned later on too, A Relying Party implementation typically consists of both a browser(client)-side and a server-side component: note “server”! “Server” in this context often means Relying Party!
  • WebAuthn RFC
  • WebAuthN Terminology
  • Client i.e. WebAuthn Client is an intermediary entity typically implemented in the user agent
  • Web Authentication API “The basic idea is that the credentials belong to the user and are managed by an authenticator, with which the WebAuthn Relying Party interacts through the client platform. Relying Party scripts can (with the user’s consent) request the browser to create a new credential”
  • Relying Party (RP) is basically your webapp i.e. the web app that relies the authentication to take place externally – it relies to the authentication information. A WebAuthn Relying Party must behave as described in §7 WebAuthn Relying Party Operations. A Relying Party implementation typically consists of both some browser(client)-side script that invokes the Web Authentication API in the client, and a server-side component that executes the Relying Party operations and other application logic. Communication between the two components MUST use HTTPS or equivalent transport security, but is otherwise beyond the scope of this specification.
  • RP ID i.e. Relying Party Identifier is valid domain string that identifies the WebAuthn Relying Party. An RP ID is based on a host’s domain name. It does not itself include a scheme or port!
  • Authenticator. The WebAuthN credentials are created and stored in a device called an authenticator that is a cryptographic entity used by a WebAuthn Client to (i) generate a public key credential and register it with a Relying Party, and (ii) authenticate by potentially verifying the user, and then cryptographically signing and returning, in the form of an Authentication Assertion, a challenge and other data presented by a WebAuthn Relying Party
  • Attestation. WebAuthn supports device Attestation. Relying Parties can request on authenticator registration that the WebAuthn Authenticator device send an Attestation certificate. The relying party can validate the attestation certificate against the publicly registered WebAuthn attestation keys, and identify the manufacturer and model of the WebAuthn Authenticator being used. By default, Attestation is not required.
  • UserAgent aka. Browser like Chrome, Firefox, etc.
  • Reminder: OpenID Connect (OIDC) has provides analogy: OpenID Provider (OP) is a front-end to OIDC Identity Providers (IdPs) but it may also act as an IdP!

Two main operations, which are described very well in the MDN documentation:

  1. Registration
  2. Authentication

In short, you should remember that

  • User Agent is basically the browser
  • Relying Party is a webapp on internet which is using externalized WebAuthN based registration and authentication. Relying Party WebAuthN implementation executes both on the browser via WebAuthN API and on the (relying party) server. See details here.
  • Authenticator is a device where credentials are stored
  • Browser support for WebAuthN

Registration

Flow (Source: auth0.com)

Authentication

Flow (Source: Auth0.com)

Last thing that might be confusing: how does a physical device communicate with a software browser? The answer to this question is in the heart of the Webauthn spec. Webauthn specification explains this:

https://www.w3.org/TR/webauthn/images/webauthn-authentication-flow-01.svg
WebAuthn Client Authentication (Source: W3C)

Relationship Between OpenID Connect and OAuth2

OAuth2 defines the following roles:

  • resource owner (RO) is an entity capable of granting access to a protected resource. When the resource owner is a person, it is referred to as an end-user.
  • resource server (RS) is the server hosting the protected resources, capable of accepting and responding to protected resource requests using access tokens.
  • client (CA) is an application making protected resource requests on behalf of the resource owner and with its authorization. The term “client” does not imply any particular implementation characteristics (e.g., whether the application executes on a server, a desktop, or other devices).
  • authorization server (AS) is the server issuing access tokens to the client after successful.

OpenID Connect has the following roles:

  • Relying Party (RP) OAuth 2.0 Client application requiring End-User Authentication and Claims from an OpenID Provider.
  • OpenID Provider (OP) OAuth 2.0 Authorization Server that is capable of Authenticating the End-User and providing Claims to a Relying Party about the Authentication event and the End-User.
OAuth2OpenID Connect
Resource (protected)A protected resource owned by a Resource Owner (you). There can be many resources of any type, but OpenID is supported by the OIDC.

This is important!
Note that IMO “openid” is NOT a protected resource but UserInfo is: Auth Code Flow returns it together with Access token, but it should be considered to be a unprotected resource whose integrity is maintained! This means that you can trust that the “openid” is correct, but also that ID Token should not be what the spec states: it should be a signed JWT with public user identifier. So basically you should not be able to explicitly choose to authorize access to “openid”: if you click a button/link that initiated OIDC authentication the “openid” should be returned if user has a valid OP session! OP must be always require user to click a button and not allow fully automatic process to prevent harvesting of public ids for malicious purposes (this is similar but different to what “prompt” with value “consent” does in the OIDC spec)! In my design, Authentication Endpoint is where user secrets (password an PIN) can be accessed, and my identity application is NOT a client (to itself) meaning when user authenticates the Authentication endpoint returns both (full) ID Token and access token (no need to exchange it then – the exchange part is thus skipped)! And actually, this authentication is not part of OIDC flow.
Resource ServerRelying Party accesses resources on behalf of the Resource owner when the Resource owner authorizes access to them. Scope attribute is used for listing the types of authorized access e.g. permissions. For example, “Facebook wants to READ your email address.” and you decide if you want to authorize the access or not.
ClientRelying Party is an Oauth2 Client
Authorization ServerOpenID Orovider (when AS supports OIDC)

Authorization is transmitted as access tokens: “If access tokens are self-contained type, access token information (such as expiration date) can be obtained by reading the content of the access tokens. Resource servers do not have to make inquiries to the introspection endpoint of the authorization server.”

In my own case. User gets ID Token (i.e. JWT token containing “openid”) when authenticating to my service. My service however also acts as a Relying Party so it gets also an access token that is used for granting right to access my protected API. However, ID Token is returned as required by the OIDC specification, but the content of ID Token could the public user identifier. As explained well: “The OpenID authentication request is essentially an OAuth 2.0 authorisation request to access the user’s identity, indicated by an openid value in the scope parameter.” The UserInfo Endpoint is an OAuth 2.0 Protected Resource that returns Claims about the authenticated End-User. So maybe the correct way of getting “openid” would be actually: a client accesses “UserInfo” resource with access token – and not getting it from the Token endpoint directly (although spec allows it…).

P.S OpenID Connect flow explained really well

RESTQL

TLDR; Divide one API into multiple (API) applications. Each application has only one endpoint for POST (C), GET (R), PUT/PATCH (U), and DELETE (D). The one endpoint is aggregated by using string encoding with slashes, for example “/apis/public/cars/id=3” and “/apis/public/cars/all” would share the endpoint of “/apis/public/cars” and text to parsed would be “id=3” or “all”. First this might seem to be a step backwards, but just like with GraphQL you actually need to write less boilerplate code – much less!

You should consider that also the GET/read is actually a write operation in the sense that the operation fetches current state and deletes any uncommitted changes. You most likely need versioning so that none of the write operations can be considered dangerous – as in permanent data loss! Versioning store every existing values into a “log” – not just the latest value, which enables checking previous values from the history…

In other posts I have described how to emulate graph database with a normal relational database like PostgreSQL. The core idea is that each entity is either a node or an edge, but both of them have properties/attributes. Links are separate tables: either unidirectional or bidirectional one-to-one, one-to-many, many-to-one, or many-to-many. Some of the objects which APIs expose are abstract meaning they don’t have a matching entity in the database.

There is the modern query languages like Elastic query language and GraphQL, but I have always felt that although they seem to be developer friendly, I usually always tend to stick with REST approach. So one day I had an idea: why does not REST have its own query language? Out of that RESTful Query Language aka RESTQL was born. So what is it then?

  • almost same as Cypher query lang for graph databases (Neo4J) but for REST. As my approach is to emulate graph databases, Cypher is pretty close to what we need!
  • similar to GraphQL but a bit different (not like https://www.graphql-java.com/ though)
  • you can query nodes and edges, and also their properties
  • RESTQL will be an extension to my previous idea where I had types which contained many different fields, and if field is NOT used it will be NULL, and the e.g. it won’t be included in the resulting JSON. NULL is needed especially in statically typed languages
  • So for example if a Book class has loaned, name, and isbn fields, an incoming JSON (RESTQL query) could be something that selects/filters an output JSON response => query1: is book loaned == /rest/book … {loaned} (book node props) query2: read node props /rest/book … see the code example below
  • one of the core benefits and trade-offs is that not everything needs to be written as a separate object when query maker i.e. client can just query data (like in GraphQL, but data is not directly exposed .. read/write models). This will be simpler version of GraphQL!!! The query name reveals the intent e.g. isBookLoaned , allBookDetails (but avoid this because then you need to write logic) Intent is partially already present in the REST API endpoint name.

Design principles

  • You should be able to query API applications and their endpoints
  • But you should also be able to query RESTful resouces
  • API part of a web application should be possible to separate a separate module
  • A web application data model relies on resources. Resources are entities which are indirectly exposed via REST API.
{
	"bookQueryX": {
		"test": null,
		"me": false,
		"this": true
	}
}

Emulating It… Part 2/2: Graph Queries With One Endpoint REST API

GraphQL was a buzzword some time ago. When I first read about it, I was fascinated. Mainly because I had been frustrated with:

  • writing too many single HTTP (REST) end-points
  • writing a lot of boilerplate code including mapping code, POJOs, DTOs, etc.
  • writing not only backend code, but writing matching code on frontend side too

So this got me thinking? Although it is not generally supported, why not to write multiple REST applications so that the actual endpoints would not be separate blocks decorated with all sorts of magical annotations or conventions. Why resource endpoint could not be identified and separated only by a String? What if an endpoint receives POST, GET, PUT/PATCH, DELETE HTTP requests, but message it self contains the resource URL as name?


# POST /apis/api/users HTTP/1.1

# Resource below is the relative to REST app path

# Virtual path is '/apis/api/users/user'

{

"method": "POST",

"resource": "/user",

"data": {

"firstName": "John",

"lastName": "Smith",

"email": "john.smith@example.com"

}

}

Emulating It… Part 1/2: Graph Database With Relational Database – And With REST too!

TLDR; If you are a experienced (web back-end) developer, you must have encountered the two following problems for which I offer a solution in this post:

  1. ORM impedance mismatch.
  2. REST API nested resources.

How to emulate a graph database with relational database? Why would you even do something like that in first place? A graph database has nodes and edges, and both of them can have properties/attributes. A node is an object and objects can have associations between them. Other members are properties. Traditionally relational database rows have been used for representing objects/entities, and foreign keys for representing relations between them. Emulation is a novel concept to extract relations into their own entities, which allows decreasing ORM impedance mismatch. In this post the “ORM” will be Java’s JPA.

Basically, each JPA entity will be just a POJO. There will be no references in these POJOs to other POJOS. And as stated, relations will be independent entities: all mappings are supported including unidirectional/bidirectional one-to-one, one-to-many, many-to-one, and many-to-many. For example, if you add a relation between two entities, it means that you will add a row into a association relation table. Relation tables resemble join-tables. And since relations are no longer object references in the source code, you can avoid complex lazy/eager logic, cascade definitions, etc. In my experience code becomes cleaner and easier to maintain (read). Yes, in my case the experience means actual source code – not spoken words.

However, there is even more! We can extend this thinking to concern REST APIs too! When you realize that everything is a resource, you can start appreciating this idea even more! Why? You can imagine that every REST resource is an abstract entity which may or may not have a matching database entity. All REST resources basically form a graph of resources where, again, links can be transformed into independent resources. This way there is no longer need for the problematic “nested resource” solution, and the simplicity emerges – trust me 🙂

Designing OAuth2 Resource Server (RS)

How to design a OAuth2 Resource Server (RS)? Do I even care – shouldn’t there be already several existing open source implementations? Well, not exactly. The only implementation I have found is Authlete. In addition, the OAuth2 specifications are not covering Resource server implementation.

Different things to consider:

In OAuth2, you grant a (client) application to access a resource in RS on behalf of the user / resource owner / you. The client needs have an access token in order to access the requested resource. The level of access is defined as scopes, which are part of the OAuth2 standards.

What does a scope mean? Oauth2 scopes are used to express what app can do on behalf of a user. They are used for handling delegation scenarios. Resource endpoints are responsible for combining the incoming scopes and actual user privileges to decide effective permissions and make access control decisions. In a way, scope is a aggregate / interface which is then implemented on RS – and not on the Authorization server (AS)! In AS, clients are managed, all available scopes are managed, and UI for allowing user to give consents are there.

Resource server access control should start with HTTP CRUD access control: i.e. POST, GET, PUT, DELETE. See my other post about organizing REST APIs! Identity of a user comes from a same domain JWT ID Token, so the user information is present. Note that scopes are client specific and they are basically transformed into access control policies! What I mean by this is that actually scopes cover only a subset of all access rights in RS! So what people usually get wrong when designing a RS is that they think that scopes are the basis of controlling access to resources. Unfortunately, I haven’t found any good resources on designing this part – have you?