Jekyll2021-07-13T15:02:56+02:00http://www.smartjava.org/feed.xmlSmartJavaArticles, books and stuff I find interestingJos DirksenWorking with nix-shell2021-07-13T00:00:00+02:002021-07-13T00:00:00+02:00http://www.smartjava.org/content/nix-shell<h1 id="managing-runtime-dependencies-using-nix">Managing runtime dependencies using Nix</h1>
<p>We’re setting up a new project at work, and wanted to apply some lessons learned from previous projects. One of the things we ran into was that when onboarding new people, or setting up a new development environment on a different machine, the dependencies were sometimes out of sync. Usually this was quite easy to fix, but keeping all the versions in sync, and dealing with deprecated or not working features (e.g <code class="highlighter-rouge">terraform</code> or <code class="highlighter-rouge">kustomize</code>) quickly becomes very annoying.
Recently we ran into another issues where a newer version of <code class="highlighter-rouge">java</code> caused issues, which led to a new version of <code class="highlighter-rouge">gradle</code>, which meant updating our build files etc. While not an issue in itself, and something that we had to do eventually, but we didn’t really planned to do this at that moment.</p>
<p>So looking around a bit we decided to look deeper into <a href="https://nixos.org/">nix</a>, and more specifically into <a href="https://nixos.org/manual/nix/unstable/command-ref/nix-shell.html">nix-shell</a>. This will allow us to create (and commit to git) a basic shell that will install all the dependencies we need, so everybody has the same starting point. Onboarding a new developer will then be just as easy as running:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>➜ nix-shell dev-env.nix
➜ { dev } nix
</code></pre></div></div>
<p><em>Note that in the previous shell, we’ve used the <a href="https://github.com/chisui/zsh-nix-shell/">zsh nix shell</a> extension, which make sure nix and zsh play nicely together. If we’re in a nix-shell I add the <code class="highlighter-rouge">{ dev }</code> prefix to indicate this</em></p>
<p>The biggest part of this setup is inspired on the repo from <a href="https://github.com/gvolpe/sbt-nix.g8">Gabriel Volpe</a> where he explains how to use it for sbt. So for more ideas and some more advanced usages check out that one as well.</p>
<h2 id="basic-setup">Basic setup</h2>
<p>So what do you need to do to get this working. The first thing to do is install nix itself. This can be easily done by following the instructions from here: https://nixos.org/download.html. Once installed you can use it using the <code class="highlighter-rouge">nix</code> command:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ nix --version
nix (Nix) 2.3.12
</code></pre></div></div>
<p>There is a whole lot learning material out there on how to use nix, also for setup and managing of your own environment, but that’s a bit out of scope for this article on nix-shell. Just a quick couple of commands to show you how interesting the approach of nix is. Whenever you install a package (or set of packages) a new state (a generation in nix terms) is created. So when we start we just have a pretty empty profile:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ nix-env --list-generations | cat
1 2021-06-16 15:49:02
2 2021-06-30 15:56:56
3 2021-06-30 15:56:56 (current)
</code></pre></div></div>
<p>So we’ve got a couple of generations, when you install it you should see something similar:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ nix-env --query "*" | cat
nix-2.3.13
nss-cacert-3.66
</code></pre></div></div>
<p>If we install a package, we’ll get a new generation, and the commands will become available. You can also remove generations, important note here, don’t remove the generations created by the install of nix. If you do this, you’ll run into strange behavior.</p>
<p>For instance lets say we want to install a specific version of java. Currently I’ve got a system wide one installed (outside of nix):</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ java -version
openjdk version "11.0.2" 2019-01-15
OpenJDK Runtime Environment 18.9 (build 11.0.2+9)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.2+9, mixed mode)
</code></pre></div></div>
<p>So let’s see what kind of versions are provided by nix, and install one. The following command will search (using a regex) through all the package descriptions that match <code class="highlighter-rouge">jdk</code>.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ nix-env -qaP --description '.*jdk.*' | cat
nixpkgs.adoptopenjdk-bin adoptopenjdk-hotspot-bin-11.0.10 AdoptOpenJDK, prebuilt OpenJDK binary
nixpkgs.adoptopenjdk-hotspot-bin-13 adoptopenjdk-hotspot-bin-13.0.2 AdoptOpenJDK, prebuilt OpenJDK binary
nixpkgs.adoptopenjdk-hotspot-bin-14 adoptopenjdk-hotspot-bin-14.0.2 AdoptOpenJDK, prebuilt OpenJDK binary
nixpkgs.adoptopenjdk-hotspot-bin-15 adoptopenjdk-hotspot-bin-15.0.2 AdoptOpenJDK, prebuilt OpenJDK binary
nixpkgs.adoptopenjdk-hotspot-bin-16 adoptopenjdk-hotspot-bin-16.0.0 AdoptOpenJDK, prebuilt OpenJDK binary
nixpkgs.adoptopenjdk-hotspot-bin-8 adoptopenjdk-hotspot-bin-8.0.282 AdoptOpenJDK, prebuilt OpenJDK binary
nixpkgs.adoptopenjdk-icedtea-web adoptopenjdk-icedtea-web-1.8.6 Java web browser plugin and an implementation of Java Web Start
nixpkgs.icedtea8_web adoptopenjdk-icedtea-web-1.8.6 Java web browser plugin and an implementation of Java Web Start
nixpkgs.icedtea_web adoptopenjdk-icedtea-web-1.8.6 Java web browser plugin and an implementation of Java Web Start
nixpkgs.adoptopenjdk-jre-bin adoptopenjdk-jre-hotspot-bin-11.0.10 AdoptOpenJDK, prebuilt OpenJDK binary
nixpkgs.adoptopenjdk-jre-hotspot-bin-13 adoptopenjdk-jre-hotspot-bin-13.0.2 AdoptOpenJDK, prebuilt OpenJDK binary
nixpkgs.adoptopenjdk-jre-hotspot-bin-14 adoptopenjdk-jre-hotspot-bin-14.0.2 AdoptOpenJDK, prebuilt OpenJDK binary
nixpkgs.adoptopenjdk-jre-hotspot-bin-15 adoptopenjdk-jre-hotspot-bin-15.0.2 AdoptOpenJDK, prebuilt OpenJDK binary
nixpkgs.adoptopenjdk-jre-hotspot-bin-16 adoptopenjdk-jre-hotspot-bin-16.0.0 AdoptOpenJDK, prebuilt OpenJDK binary
nixpkgs.adoptopenjdk-jre-hotspot-bin-8 adoptopenjdk-jre-hotspot-bin-8.0.282 AdoptOpenJDK, prebuilt OpenJDK binary
nixpkgs.adoptopenjdk-jre-openj9-bin-11 adoptopenjdk-jre-openj9-bin-11.0.10 AdoptOpenJDK, prebuilt OpenJDK binary
nixpkgs.adoptopenjdk-jre-openj9-bin-13 adoptopenjdk-jre-openj9-bin-13.0.2 AdoptOpenJDK, prebuilt OpenJDK binary
nixpkgs.adoptopenjdk-jre-openj9-bin-14 adoptopenjdk-jre-openj9-bin-14.0.2 AdoptOpenJDK, prebuilt OpenJDK binary
nixpkgs.adoptopenjdk-jre-openj9-bin-15 adoptopenjdk-jre-openj9-bin-15.0.2 AdoptOpenJDK, prebuilt OpenJDK binary
nixpkgs.adoptopenjdk-jre-openj9-bin-16 adoptopenjdk-jre-openj9-bin-16.0.0 AdoptOpenJDK, prebuilt OpenJDK binary
nixpkgs.adoptopenjdk-jre-openj9-bin-8 adoptopenjdk-jre-openj9-bin-8.0.282 AdoptOpenJDK, prebuilt OpenJDK binary
nixpkgs.adoptopenjdk-openj9-bin-11 adoptopenjdk-openj9-bin-11.0.10 AdoptOpenJDK, prebuilt OpenJDK binary
nixpkgs.adoptopenjdk-openj9-bin-13 adoptopenjdk-openj9-bin-13.0.2 AdoptOpenJDK, prebuilt OpenJDK binary
nixpkgs.adoptopenjdk-openj9-bin-14 adoptopenjdk-openj9-bin-14.0.2 AdoptOpenJDK, prebuilt OpenJDK binary
nixpkgs.adoptopenjdk-openj9-bin-15 adoptopenjdk-openj9-bin-15.0.2 AdoptOpenJDK, prebuilt OpenJDK binary
nixpkgs.adoptopenjdk-openj9-bin-16 adoptopenjdk-openj9-bin-16.0.0 AdoptOpenJDK, prebuilt OpenJDK binary
nixpkgs.adoptopenjdk-openj9-bin-8 adoptopenjdk-openj9-bin-8.0.282 AdoptOpenJDK, prebuilt OpenJDK binary
nixpkgs.jetbrains.jdk jetbrains-jdk-11.0.10-b1427 An OpenJDK fork to better support Jetbrains's products.
nixpkgs.oraclejdk11 oraclejdk-11.0.10
nixpkgs.oraclejdk14 oraclejdk-14.0.2
nixpkgs.oraclejdk8 oraclejdk-8u281
nixpkgs.oraclejdk oraclejdk-8u281
nixpkgs.jdk11 zulu11.48.21-ca-jdk-11.0.11 The open-source Java Development Kit
nixpkgs.jdk zulu16.30.15-ca-jdk-16.0.1 The open-source Java Development Kit
nixpkgs.jre_minimal zulu16.30.15-ca-jdk-16.0.1-minimal-jre
nixpkgs.jre8 zulu8.54.0.21-ca-jdk-8.0.292 The open-source Java Development Kit
nixpkgs.jdk8 zulu8.54.0.21-ca-jdk-8.0.292 The open-source Java Development Kit
</code></pre></div></div>
<p>Here we see the package name, which you use to install the package, a human readable name, and some description. You can also search using the <code class="highlighter-rouge">nix</code> command tool. This provides pretty much the same results, but adds a bit more human readable output.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ nix search jdk | cat
* nixpkgs.adoptopenjdk-bin (adoptopenjdk-hotspot-bin)
AdoptOpenJDK, prebuilt OpenJDK binary
* nixpkgs.adoptopenjdk-hotspot-bin-11 (adoptopenjdk-hotspot-bin)
AdoptOpenJDK, prebuilt OpenJDK binary
* nixpkgs.adoptopenjdk-hotspot-bin-13 (adoptopenjdk-hotspot-bin)
AdoptOpenJDK, prebuilt OpenJDK binary
</code></pre></div></div>
<p>And finally you can also search online (https://search.nixos.org/packages), but it is always good to at least know some of the command line tools you can use. So from the list above lets say that currently we’re interested in installing jdk-15, just because we can.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ nix-env -i adoptopenjdk-hotspot-bin-15.0.2
</code></pre></div></div>
<p>This will kick off the installation of this package, and any dependencies it might have. Once done we’ll have an additional generation:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ nix-env --list-generations
1 2021-06-16 15:49:02
2 2021-06-30 15:56:56
3 2021-06-30 15:56:56
4 2021-06-30 16:01:57 (current)
</code></pre></div></div>
<p>We can see what we installed:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ nix-env --query "*" | cat
adoptopenjdk-hotspot-bin-15.0.2
nix-2.3.13
nss-cacert-3.66
</code></pre></div></div>
<p>And we’ve got the expected version of java:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ java -version
openjdk version "15.0.2" 2021-01-19
</code></pre></div></div>
<p>But, wait, maybe you made an error and didn’t really want to use openjdk-15. Then you can just do a rollback, and you go back to the exact state your system was in before you installed the new libraries and tools that led to this version:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ nix-env --rollback
switching from generation 4 to 3
$ java -version
openjdk version "11.0.2" 2019-01-15
OpenJDK Runtime Environment 18.9 (build 11.0.2+9)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.2+9, mixed mode)
</code></pre></div></div>
<p>And if you want to switch back to generation 4, you can also do that:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ nix-env --switch-generation 4
$ java -version
openjdk version "15.0.2" 2021-01-19
</code></pre></div></div>
<p>Easy right? So you can very easily setup your environment, experiment with new packages and tools, without having to worry that you might break your system.</p>
<h2 id="nix-shell">nix-shell</h2>
<p>In the previous setup we just showed you how to setup your own environment, your own shell. Often, though, when you start a new project, or switch to some legacy repository you need to work on, you require specific tools and versions to be able to work with that setup. It might need an older version of <code class="highlighter-rouge">gradle</code>, or hasn’t been updated to the latest <code class="highlighter-rouge">java</code> version, or any of the other hundreds of reasons why specific versions are needed. For this nix comes with <code class="highlighter-rouge">nix-shell</code>. With nix-shell you can parse a configuration file written in <code class="highlighter-rouge">nix</code> (https://nixos.org/guides/nix-pills/basics-of-language.html), and get a new shell that has installed exactly what was defined in the script. So when you start something new, or someone needs to be onboarded, all they have to do to get started (without messing up their laptops and running into conflicts) is install nix, and run <code class="highlighter-rouge">nix-shell env.nix</code>, where <code class="highlighter-rouge">env-nix</code> is the configuration of your environment.</p>
<p>To set this up, we define a script written in nix, that defines the packages (and other stuff) that we need to have in our environment (once again, look at the stuff from https://github.com/gvolpe/sbt-nix.g8 where most of this is based on). We’ll first look at what happens when we run this script, and then what’s in it.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ nix-shell dev-env.nix
unpacking 'https://github.com/NixOS/nixpkgs/archive/189a13688782.tar.gz'...
these paths will be fetched (462.10 MiB download, 1343.75 MiB unpacked):
/nix/store/0w0hm71xblin82k00mlb7b8dsnf9qxl6-hook
/nix/store/2s9yf506ma88n1flxnv10swv6n5klfzf-zulu16.30.15-ca-jdk-16.0.1
/nix/store/34r722s1g3wx73v6prmx0djaq0gw95p3-jq-1.6-dev
/nix/store/3q5c3nvgmd5h0s7h1gks30b009g68ki1-terraform-1.0.0
/nix/store/4gkpfv7x4qdchwfalsb8hh2q62x9l6nl-kustomize-4.1.3
/nix/store/5im8ywgl0cilxlysvrwdnhik954nivbm-clang-7.1.0
/nix/store/64d69jqbz4s8ziqbpam41sd70w338ars-libcxx-7.1.0
...
</code></pre></div></div>
<p>What happens here is that all the dependencies we’ve defined in the <code class="highlighter-rouge">dev-env.nix</code> script are downloaded, and installed for this local shell. They won’t interfere with anything else you’ve got running or if you’re using nix standalone as well. In our case we installed specific versions (which are installed the first time you run this) of <code class="highlighter-rouge">java</code>, <code class="highlighter-rouge">jq</code>, <code class="highlighter-rouge">terraform</code>, <code class="highlighter-rouge">gradlew</code> and <code class="highlighter-rouge">kustomize</code>.</p>
<p>Once this is done running, we’ve got a shell, where these specific versions are available, without interfering with the rest of the OS and any packages installed globally. To set this up we use a number of different files. The first one we’ll look at is the <code class="highlighter-rouge">dev-env.nix</code> file:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>{ jdk ? "jdk16" }:
let
# get a normalized set of packages, from which
# we will install all the needed dependencies
pkgs = import ./pkgs.nix { inherit jdk; };
in
pkgs.mkShell {
buildInputs = [
pkgs.${jdk}
pkgs.gradle
pkgs.jq
pkgs.kubectl
pkgs.kustomize
pkgs.terraform_1_0
];
shellHook = ''
export NIX_ENV=dev
'';
}
</code></pre></div></div>
<p>Here we specify with the <code class="highlighter-rouge">mkShell</code> function which packages we want to have available. At the top is an argument to this shell, so we can override the JDK version should we want to do that. If we don’t want this, then we’ll get <code class="highlighter-rouge">jdk16</code>. The goal is to have a reproducible environment, so we need a way to fix the packages. In <code class="highlighter-rouge">nix</code> we can do this by pointing to a specific git commit of where we want to get the package list from. For this we need to look at the <code class="highlighter-rouge">pkgs.nix</code> file next:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>{ jdk }:
let
pinned = import ./pinned.nix;
config = import ./config.nix { inherit jdk; };
# if we want to configure a new version of terraform which isn't available
# yet, we could use an overlay. eg.
# overlays = [
# (import ./terraform.nix)
#];
# pkgs = import pinned.nixpkgs { inherit config; inherit overlays;};
pkgs = import pinned.nixpkgs { inherit config;};
in
pkgs
</code></pre></div></div>
<p>Here we configure the <code class="highlighter-rouge">pkgs</code> variable, which will contain all the packages and settings which are available to install from. As you can see from this short script, we define a <code class="highlighter-rouge">pkgs</code> variable which is based on the information from <code class="highlighter-rouge">pinned</code>. Which in itself is read from <code class="highlighter-rouge">pinned.nix</code>. Before we look at the specific <code class="highlighter-rouge">config.nix</code> we’ll first look at the <code class="highlighter-rouge">pinned.nix</code> file:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>{
# We fix to a specific nixos version.
nixpkgs = fetchTarball {
name = "nixos-unstable-2021-06-172";
url = "https://github.com/NixOS/nixpkgs/archive/189a13688782.tar.gz";
sha256 = "09a027w36x05c8m8rwa7lr2g4sc12hx502xbkxhrpa3vmcryrc51";
};
}
</code></pre></div></div>
<p>What this means is that we define the <code class="highlighter-rouge">nixpkgs</code> variable to a specific archive, which can be downloaded from that location. This is just the total list of packages available, including version information, that was available at that time. Since everyone will use this same version, we can assure that everyone gets the same dependencies. There are different initiatives that aim in making working with specific versions easier:</p>
<ul>
<li><a href="https://github.com/nmattia/niv">niv</a>: Niv stores the specific versions in a separate json file, and seems like a good approach.</li>
<li><a href="https://nix.dev/tutorials/towards-reproducibility-pinning-nixpkgs">nix-dev</a>: Provides additional information and resources on how to use nix for development.</li>
</ul>
<p>Now we have a look at the <code class="highlighter-rouge">config.nix</code> (where we pass in the <code class="highlighter-rouge">jdk</code> as a parameter):</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>{ jdk }:
{
packageOverrides = p: {
gradle = (p.gradleGen.override {
java = p.${jdk};
}).gradle_latest;
};
}
</code></pre></div></div>
<p>Here we define the <code class="highlighter-rouge">packageOverride</code>, where we fix the java version of <code class="highlighter-rouge">gradle</code> to the version that we want. So when we install <code class="highlighter-rouge">pkgs.gradle</code>, this configuration will be used. If we got more <code class="highlighter-rouge">packageOverrides</code> we can add them here for our specific configurations. Note here that <code class="highlighter-rouge">packageOverrides</code> can only be set once, and don’t compose like <a href="https://nixos.wiki/wiki/Overlays">overlays</a> do. But for this example, it’s an easy way to configure <code class="highlighter-rouge">gradle</code>.</p>
<p>As an example of how this would work in an overlay, we can define one for a custom version of <code class="highlighter-rouge">terraform</code> like this:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code># Example of an overlay, where we replace the default from nix, with
# a new description.
final: prev: {
terraform = prev.terraform_1_0.overrideAttrs (old: rec {
version = "1.0.0";
name = "terraform-${version}";
src = prev.fetchFromGitHub {
owner = "hashicorp";
repo = "terraform";
rev = "v${version}";
sha256 = "sha256-ddcT/I2Qn1pKFyhXgh+CcD3fSv2steSNmjyyiS2SE/o=";
};
});
}
</code></pre></div></div>
<p>And in the <code class="highlighter-rouge">pkgs.nix</code> you can see how we can also add overlays. And as you can see, overlays is provided as an array, while the <code class="highlighter-rouge">packageOverrides</code> is a single argument to <code class="highlighter-rouge">nixpkgs</code>.</p>
<p>In the end, with this configuration we’ve got a nice basic setup to fix development (or build) tools to specific versions, which can easily be shared in git. Using this will allow new people to be quickly onboarded, without having to find and install specific versions, which might conflict with already installed tools.</p>
<h2 id="wrap-up">wrap up</h2>
<p>What was shown in this article is a very small set of what nix is capable of. You can easily create specific environments for docker, for building, or for different development environments. And while in this example we used <code class="highlighter-rouge">nix-shell</code> to setup a simple development environment, you can do much more. You can for instance use it to create a default home shell on your system, which you can quickly provision whenever you switch machines, or need to setup a new system: <a href="https://github.com/nix-community/home-manager">Nix Home Manager</a></p>
<p>A couple of other nice articles on this:</p>
<ul>
<li>https://ghedam.at/15978/an-introduction-to-nix-shell</li>
<li>https://myme.no/posts/2020-01-26-nixos-for-development.html</li>
</ul>Jos DirksenManaging runtime dependencies using Nix We’re setting up a new project at work, and wanted to apply some lessons learned from previous projects. One of the things we ran into was that when onboarding new people, or setting up a new development environment on a different machine, the dependencies were sometimes out of sync. Usually this was quite easy to fix, but keeping all the versions in sync, and dealing with deprecated or not working features (e.g terraform or kustomize) quickly becomes very annoying. Recently we ran into another issues where a newer version of java caused issues, which led to a new version of gradle, which meant updating our build files etc. While not an issue in itself, and something that we had to do eventually, but we didn’t really planned to do this at that moment. So looking around a bit we decided to look deeper into nix, and more specifically into nix-shell. This will allow us to create (and commit to git) a basic shell that will install all the dependencies we need, so everybody has the same starting point. Onboarding a new developer will then be just as easy as running: ➜ nix-shell dev-env.nix ➜ { dev } nix Note that in the previous shell, we’ve used the zsh nix shell extension, which make sure nix and zsh play nicely together. If we’re in a nix-shell I add the { dev } prefix to indicate this The biggest part of this setup is inspired on the repo from Gabriel Volpe where he explains how to use it for sbt. So for more ideas and some more advanced usages check out that one as well. Basic setup So what do you need to do to get this working. The first thing to do is install nix itself. This can be easily done by following the instructions from here: https://nixos.org/download.html. Once installed you can use it using the nix command: $ nix --version nix (Nix) 2.3.12 There is a whole lot learning material out there on how to use nix, also for setup and managing of your own environment, but that’s a bit out of scope for this article on nix-shell. Just a quick couple of commands to show you how interesting the approach of nix is. Whenever you install a package (or set of packages) a new state (a generation in nix terms) is created. So when we start we just have a pretty empty profile: $ nix-env --list-generations | cat 1 2021-06-16 15:49:02 2 2021-06-30 15:56:56 3 2021-06-30 15:56:56 (current) So we’ve got a couple of generations, when you install it you should see something similar: $ nix-env --query "*" | cat nix-2.3.13 nss-cacert-3.66 If we install a package, we’ll get a new generation, and the commands will become available. You can also remove generations, important note here, don’t remove the generations created by the install of nix. If you do this, you’ll run into strange behavior. For instance lets say we want to install a specific version of java. Currently I’ve got a system wide one installed (outside of nix): $ java -version openjdk version "11.0.2" 2019-01-15 OpenJDK Runtime Environment 18.9 (build 11.0.2+9) OpenJDK 64-Bit Server VM 18.9 (build 11.0.2+9, mixed mode) So let’s see what kind of versions are provided by nix, and install one. The following command will search (using a regex) through all the package descriptions that match jdk. $ nix-env -qaP --description '.*jdk.*' | cat nixpkgs.adoptopenjdk-bin adoptopenjdk-hotspot-bin-11.0.10 AdoptOpenJDK, prebuilt OpenJDK binary nixpkgs.adoptopenjdk-hotspot-bin-13 adoptopenjdk-hotspot-bin-13.0.2 AdoptOpenJDK, prebuilt OpenJDK binary nixpkgs.adoptopenjdk-hotspot-bin-14 adoptopenjdk-hotspot-bin-14.0.2 AdoptOpenJDK, prebuilt OpenJDK binary nixpkgs.adoptopenjdk-hotspot-bin-15 adoptopenjdk-hotspot-bin-15.0.2 AdoptOpenJDK, prebuilt OpenJDK binary nixpkgs.adoptopenjdk-hotspot-bin-16 adoptopenjdk-hotspot-bin-16.0.0 AdoptOpenJDK, prebuilt OpenJDK binary nixpkgs.adoptopenjdk-hotspot-bin-8 adoptopenjdk-hotspot-bin-8.0.282 AdoptOpenJDK, prebuilt OpenJDK binary nixpkgs.adoptopenjdk-icedtea-web adoptopenjdk-icedtea-web-1.8.6 Java web browser plugin and an implementation of Java Web Start nixpkgs.icedtea8_web adoptopenjdk-icedtea-web-1.8.6 Java web browser plugin and an implementation of Java Web Start nixpkgs.icedtea_web adoptopenjdk-icedtea-web-1.8.6 Java web browser plugin and an implementation of Java Web Start nixpkgs.adoptopenjdk-jre-bin adoptopenjdk-jre-hotspot-bin-11.0.10 AdoptOpenJDK, prebuilt OpenJDK binary nixpkgs.adoptopenjdk-jre-hotspot-bin-13 adoptopenjdk-jre-hotspot-bin-13.0.2 AdoptOpenJDK, prebuilt OpenJDK binary nixpkgs.adoptopenjdk-jre-hotspot-bin-14 adoptopenjdk-jre-hotspot-bin-14.0.2 AdoptOpenJDK, prebuilt OpenJDK binary nixpkgs.adoptopenjdk-jre-hotspot-bin-15 adoptopenjdk-jre-hotspot-bin-15.0.2 AdoptOpenJDK, prebuilt OpenJDK binary nixpkgs.adoptopenjdk-jre-hotspot-bin-16 adoptopenjdk-jre-hotspot-bin-16.0.0 AdoptOpenJDK, prebuilt OpenJDK binary nixpkgs.adoptopenjdk-jre-hotspot-bin-8 adoptopenjdk-jre-hotspot-bin-8.0.282 AdoptOpenJDK, prebuilt OpenJDK binary nixpkgs.adoptopenjdk-jre-openj9-bin-11 adoptopenjdk-jre-openj9-bin-11.0.10 AdoptOpenJDK, prebuilt OpenJDK binary nixpkgs.adoptopenjdk-jre-openj9-bin-13 adoptopenjdk-jre-openj9-bin-13.0.2 AdoptOpenJDK, prebuilt OpenJDK binary nixpkgs.adoptopenjdk-jre-openj9-bin-14 adoptopenjdk-jre-openj9-bin-14.0.2 AdoptOpenJDK, prebuilt OpenJDK binary nixpkgs.adoptopenjdk-jre-openj9-bin-15 adoptopenjdk-jre-openj9-bin-15.0.2 AdoptOpenJDK, prebuilt OpenJDK binary nixpkgs.adoptopenjdk-jre-openj9-bin-16 adoptopenjdk-jre-openj9-bin-16.0.0 AdoptOpenJDK, prebuilt OpenJDK binary nixpkgs.adoptopenjdk-jre-openj9-bin-8 adoptopenjdk-jre-openj9-bin-8.0.282 AdoptOpenJDK, prebuilt OpenJDK binary nixpkgs.adoptopenjdk-openj9-bin-11 adoptopenjdk-openj9-bin-11.0.10 AdoptOpenJDK, prebuilt OpenJDK binary nixpkgs.adoptopenjdk-openj9-bin-13 adoptopenjdk-openj9-bin-13.0.2 AdoptOpenJDK, prebuilt OpenJDK binary nixpkgs.adoptopenjdk-openj9-bin-14 adoptopenjdk-openj9-bin-14.0.2 AdoptOpenJDK, prebuilt OpenJDK binary nixpkgs.adoptopenjdk-openj9-bin-15 adoptopenjdk-openj9-bin-15.0.2 AdoptOpenJDK, prebuilt OpenJDK binary nixpkgs.adoptopenjdk-openj9-bin-16 adoptopenjdk-openj9-bin-16.0.0 AdoptOpenJDK, prebuilt OpenJDK binary nixpkgs.adoptopenjdk-openj9-bin-8 adoptopenjdk-openj9-bin-8.0.282 AdoptOpenJDK, prebuilt OpenJDK binary nixpkgs.jetbrains.jdk jetbrains-jdk-11.0.10-b1427 An OpenJDK fork to better support Jetbrains's products. nixpkgs.oraclejdk11 oraclejdk-11.0.10 nixpkgs.oraclejdk14 oraclejdk-14.0.2 nixpkgs.oraclejdk8 oraclejdk-8u281 nixpkgs.oraclejdk oraclejdk-8u281 nixpkgs.jdk11 zulu11.48.21-ca-jdk-11.0.11 The open-source Java Development Kit nixpkgs.jdk zulu16.30.15-ca-jdk-16.0.1 The open-source Java Development Kit nixpkgs.jre_minimal zulu16.30.15-ca-jdk-16.0.1-minimal-jre nixpkgs.jre8 zulu8.54.0.21-ca-jdk-8.0.292 The open-source Java Development Kit nixpkgs.jdk8 zulu8.54.0.21-ca-jdk-8.0.292 The open-source Java Development Kit Here we see the package name, which you use to install the package, a human readable name, and some description. You can also search using the nix command tool. This provides pretty much the same results, but adds a bit more human readable output. $ nix search jdk | cat * nixpkgs.adoptopenjdk-bin (adoptopenjdk-hotspot-bin) AdoptOpenJDK, prebuilt OpenJDK binary * nixpkgs.adoptopenjdk-hotspot-bin-11 (adoptopenjdk-hotspot-bin) AdoptOpenJDK, prebuilt OpenJDK binary * nixpkgs.adoptopenjdk-hotspot-bin-13 (adoptopenjdk-hotspot-bin) AdoptOpenJDK, prebuilt OpenJDK binary And finally you can also search online (https://search.nixos.org/packages), but it is always good to at least know some of the command line tools you can use. So from the list above lets say that currently we’re interested in installing jdk-15, just because we can. $ nix-env -i adoptopenjdk-hotspot-bin-15.0.2 This will kick off the installation of this package, and any dependencies it might have. Once done we’ll have an additional generation: $ nix-env --list-generations 1 2021-06-16 15:49:02 2 2021-06-30 15:56:56 3 2021-06-30 15:56:56 4 2021-06-30 16:01:57 (current) We can see what we installed: $ nix-env --query "*" | cat adoptopenjdk-hotspot-bin-15.0.2 nix-2.3.13 nss-cacert-3.66 And we’ve got the expected version of java: $ java -version openjdk version "15.0.2" 2021-01-19 But, wait, maybe you made an error and didn’t really want to use openjdk-15. Then you can just do a rollback, and you go back to the exact state your system was in before you installed the new libraries and tools that led to this version: $ nix-env --rollback switching from generation 4 to 3 $ java -version openjdk version "11.0.2" 2019-01-15 OpenJDK Runtime Environment 18.9 (build 11.0.2+9) OpenJDK 64-Bit Server VM 18.9 (build 11.0.2+9, mixed mode) And if you want to switch back to generation 4, you can also do that: $ nix-env --switch-generation 4 $ java -version openjdk version "15.0.2" 2021-01-19 Easy right? So you can very easily setup your environment, experiment with new packages and tools, without having to worry that you might break your system. nix-shell In the previous setup we just showed you how to setup your own environment, your own shell. Often, though, when you start a new project, or switch to some legacy repository you need to work on, you require specific tools and versions to be able to work with that setup. It might need an older version of gradle, or hasn’t been updated to the latest java version, or any of the other hundreds of reasons why specific versions are needed. For this nix comes with nix-shell. With nix-shell you can parse a configuration file written in nix (https://nixos.org/guides/nix-pills/basics-of-language.html), and get a new shell that has installed exactly what was defined in the script. So when you start something new, or someone needs to be onboarded, all they have to do to get started (without messing up their laptops and running into conflicts) is install nix, and run nix-shell env.nix, where env-nix is the configuration of your environment. To set this up, we define a script written in nix, that defines the packages (and other stuff) that we need to have in our environment (once again, look at the stuff from https://github.com/gvolpe/sbt-nix.g8 where most of this is based on). We’ll first look at what happens when we run this script, and then what’s in it. $ nix-shell dev-env.nix unpacking 'https://github.com/NixOS/nixpkgs/archive/189a13688782.tar.gz'... these paths will be fetched (462.10 MiB download, 1343.75 MiB unpacked): /nix/store/0w0hm71xblin82k00mlb7b8dsnf9qxl6-hook /nix/store/2s9yf506ma88n1flxnv10swv6n5klfzf-zulu16.30.15-ca-jdk-16.0.1 /nix/store/34r722s1g3wx73v6prmx0djaq0gw95p3-jq-1.6-dev /nix/store/3q5c3nvgmd5h0s7h1gks30b009g68ki1-terraform-1.0.0 /nix/store/4gkpfv7x4qdchwfalsb8hh2q62x9l6nl-kustomize-4.1.3 /nix/store/5im8ywgl0cilxlysvrwdnhik954nivbm-clang-7.1.0 /nix/store/64d69jqbz4s8ziqbpam41sd70w338ars-libcxx-7.1.0 ... What happens here is that all the dependencies we’ve defined in the dev-env.nix script are downloaded, and installed for this local shell. They won’t interfere with anything else you’ve got running or if you’re using nix standalone as well. In our case we installed specific versions (which are installed the first time you run this) of java, jq, terraform, gradlew and kustomize. Once this is done running, we’ve got a shell, where these specific versions are available, without interfering with the rest of the OS and any packages installed globally. To set this up we use a number of different files. The first one we’ll look at is the dev-env.nix file: { jdk ? "jdk16" }: let # get a normalized set of packages, from which # we will install all the needed dependencies pkgs = import ./pkgs.nix { inherit jdk; }; in pkgs.mkShell { buildInputs = [ pkgs.${jdk} pkgs.gradle pkgs.jq pkgs.kubectl pkgs.kustomize pkgs.terraform_1_0 ]; shellHook = '' export NIX_ENV=dev ''; } Here we specify with the mkShell function which packages we want to have available. At the top is an argument to this shell, so we can override the JDK version should we want to do that. If we don’t want this, then we’ll get jdk16. The goal is to have a reproducible environment, so we need a way to fix the packages. In nix we can do this by pointing to a specific git commit of where we want to get the package list from. For this we need to look at the pkgs.nix file next: { jdk }: let pinned = import ./pinned.nix; config = import ./config.nix { inherit jdk; }; # if we want to configure a new version of terraform which isn't available # yet, we could use an overlay. eg. # overlays = [ # (import ./terraform.nix) #]; # pkgs = import pinned.nixpkgs { inherit config; inherit overlays;}; pkgs = import pinned.nixpkgs { inherit config;}; in pkgs Here we configure the pkgs variable, which will contain all the packages and settings which are available to install from. As you can see from this short script, we define a pkgs variable which is based on the information from pinned. Which in itself is read from pinned.nix. Before we look at the specific config.nix we’ll first look at the pinned.nix file: { # We fix to a specific nixos version. nixpkgs = fetchTarball { name = "nixos-unstable-2021-06-172"; url = "https://github.com/NixOS/nixpkgs/archive/189a13688782.tar.gz"; sha256 = "09a027w36x05c8m8rwa7lr2g4sc12hx502xbkxhrpa3vmcryrc51"; }; } What this means is that we define the nixpkgs variable to a specific archive, which can be downloaded from that location. This is just the total list of packages available, including version information, that was available at that time. Since everyone will use this same version, we can assure that everyone gets the same dependencies. There are different initiatives that aim in making working with specific versions easier: niv: Niv stores the specific versions in a separate json file, and seems like a good approach. nix-dev: Provides additional information and resources on how to use nix for development. Now we have a look at the config.nix (where we pass in the jdk as a parameter): { jdk }: { packageOverrides = p: { gradle = (p.gradleGen.override { java = p.${jdk}; }).gradle_latest; }; } Here we define the packageOverride, where we fix the java version of gradle to the version that we want. So when we install pkgs.gradle, this configuration will be used. If we got more packageOverrides we can add them here for our specific configurations. Note here that packageOverrides can only be set once, and don’t compose like overlays do. But for this example, it’s an easy way to configure gradle. As an example of how this would work in an overlay, we can define one for a custom version of terraform like this: # Example of an overlay, where we replace the default from nix, with # a new description. final: prev: { terraform = prev.terraform_1_0.overrideAttrs (old: rec { version = "1.0.0"; name = "terraform-${version}"; src = prev.fetchFromGitHub { owner = "hashicorp"; repo = "terraform"; rev = "v${version}"; sha256 = "sha256-ddcT/I2Qn1pKFyhXgh+CcD3fSv2steSNmjyyiS2SE/o="; }; }); } And in the pkgs.nix you can see how we can also add overlays. And as you can see, overlays is provided as an array, while the packageOverrides is a single argument to nixpkgs. In the end, with this configuration we’ve got a nice basic setup to fix development (or build) tools to specific versions, which can easily be shared in git. Using this will allow new people to be quickly onboarded, without having to find and install specific versions, which might conflict with already installed tools. wrap up What was shown in this article is a very small set of what nix is capable of. You can easily create specific environments for docker, for building, or for different development environments. And while in this example we used nix-shell to setup a simple development environment, you can do much more. You can for instance use it to create a default home shell on your system, which you can quickly provision whenever you switch machines, or need to setup a new system: Nix Home Manager A couple of other nice articles on this: https://ghedam.at/15978/an-introduction-to-nix-shell https://myme.no/posts/2020-01-26-nixos-for-development.htmlExploring Cats Effect2021-06-14T00:00:00+02:002021-06-14T00:00:00+02:00http://www.smartjava.org/content/exploring-cats-effect<p>In two previous articles we explored how you can use <a href="https://zio.dev">ZIO</a> to easily create programs in a type-safe manner:</p>
<ul>
<li><a href="https://www.smartjava.org/content/exploring-zio-part-1/">Exploring ZIO - Part I</a></li>
<li><a href="https://www.smartjava.org/content/exploring-zio-part-2/">Exploring ZIO - Part II - ZStream and modules</a></li>
</ul>
<p>In those articles we created a simple server which polls a REST endpoint at a fixed interval, and wrote the results in a database (MongoDB). The data was then exposed through an <a href="https://http4s.org/">HTTP4S</a> REST endpoint. In that approach we used the ZIO monad throughout the stack, which allowed us to use (in those examples only a small number) of specific ZIO features to make our lives easier. If you want to jump directly to the code you can find that here: https://github.com/josdirksen/tagless-playground</p>
<p>Another common approach for creating modules and services is by using the “Tagless Final” pattern. If you search around you’ll find numerous explanations about what this pattern does, but the main idea behind this pattern is:</p>
<ul>
<li>Defer choosing a IO implementation. So our services and traits are defined using <code class="highlighter-rouge">F[_]</code> as the type.</li>
<li>Dependencies for a certain function (or module) are defined as implicit parameters. E.g like <code class="highlighter-rouge">F[_] : ConfigDSL</code>, means that this function (or case class) requires a type <code class="highlighter-rouge">F[_]</code> for which a <code class="highlighter-rouge">ConfigDSL[F]</code> implementation is in the implicit scope.</li>
</ul>
<p>There are a number of other features in this pattern, but the above captures most of this. There are a couple of libraries that adopt this pattern, where dependencies need to be provided through implicits, and many of those use the generic <a href="https://typelevel.org/cats-effect">Cats Effect</a> library to define fine grained effects that functions need.</p>
<p>In this article we’ll convert the setup from <a href="https://www.smartjava.org/content/exploring-zio-part-2/">Exploring ZIO - Part II - ZStream and modules</a> to one where we use Cats Effect together with (part of) the “Tagless Final” pattern.</p>
<p>Before we start a quick summary of what we’re going to build:</p>
<ol>
<li>Read a configuration file using <a href="https://github.com/pureconfig/pureconfig">PureConfig</a></li>
<li>Use this configuration to create a <a href="https://http4s.org/">HTTP4S server and HTTP4S client</a></li>
<li>Create a stream using <a href="https://fs2.io">FS2</a> which is used to get the latest temperatures from a remote server.</li>
<li>Store the results from the external client in MongoDB.</li>
</ol>
<p>All the steps above will be defined using ‘Cats Effect’ and we’ll only provide a concrete IO monad implementation at the edge of the world in our <code class="highlighter-rouge">main</code> function.</p>
<h2 id="loading-configuration">Loading configuration</h2>
<p>The main difference with the ZIO approach is that we define the trait using a <code class="highlighter-rouge">F[_]</code></p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">case</span> <span class="k">class</span> <span class="nc">Config</span><span class="o">(</span><span class="n">apiConfig</span><span class="k">:</span> <span class="kt">ApiConfig</span><span class="o">,</span> <span class="n">temperatureConfig</span><span class="k">:</span> <span class="kt">TemperatureConfig</span><span class="o">,</span> <span class="n">dbConfig</span><span class="k">:</span> <span class="kt">DBConfig</span><span class="o">)</span>
<span class="k">case</span> <span class="k">class</span> <span class="nc">ApiConfig</span><span class="o">(</span><span class="n">endpoint</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">port</span><span class="k">:</span> <span class="kt">Int</span><span class="o">)</span>
<span class="k">case</span> <span class="k">class</span> <span class="nc">TemperatureConfig</span><span class="o">(</span><span class="n">endpoint</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">apiKey</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">interval</span><span class="k">:</span> <span class="kt">FiniteDuration</span><span class="o">)</span>
<span class="k">case</span> <span class="k">class</span> <span class="nc">DBConfig</span><span class="o">(</span><span class="n">endpoint</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span>
<span class="cm">/**
* Basic DSL trait is type, constraint agnostic.
*/</span>
<span class="k">trait</span> <span class="nc">ConfigDSL</span><span class="o">[</span><span class="kt">F</span><span class="o">[</span><span class="k">_</span><span class="o">]]</span> <span class="o">{</span>
<span class="k">val</span> <span class="n">config</span><span class="k">:</span> <span class="kt">F</span><span class="o">[</span><span class="kt">Config</span><span class="o">]</span>
<span class="o">}</span>
<span class="k">object</span> <span class="nc">ConfigDSL</span> <span class="o">{</span>
<span class="k">def</span> <span class="n">apply</span><span class="o">[</span><span class="kt">F</span><span class="o">[</span><span class="k">_</span><span class="o">]](</span><span class="k">implicit</span> <span class="n">F</span><span class="k">:</span> <span class="kt">ConfigDSL</span><span class="o">[</span><span class="kt">F</span><span class="o">])</span><span class="k">:</span> <span class="kt">F.</span><span class="k">type</span> <span class="o">=</span> <span class="n">F</span>
<span class="o">}</span>
</code></pre></div></div>
<p>To use this config we’re going to define an interpreter for it. While we can have multiple interpreters, we should preferably only have one. A simple implementation using PureConfig is shown here. Note that in this implementation we still don’t decide on the actual IO monad we’re going to run the final program in. That’s something we decide at the latest possible time.</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cm">/**
* A pureconfig based interpreter. Note that we use the kind projector, so we can
* correctly specify the MonadError constraint. If we don't use this we have to
* either add it as a desugared implicit, or create a custom type.
*/</span>
<span class="k">case</span> <span class="k">class</span> <span class="nc">LiveConfigInterpreter</span><span class="o">[</span><span class="kt">F</span><span class="o">[</span><span class="k">_</span><span class="o">]</span><span class="kt">:</span> <span class="kt">MonadError</span><span class="o">[</span><span class="kt">*</span><span class="o">[</span><span class="k">_</span><span class="o">]</span>, <span class="kt">Throwable</span><span class="o">]]()</span> <span class="k">extends</span> <span class="nc">ConfigDSL</span><span class="o">[</span><span class="kt">F</span><span class="o">]</span> <span class="o">{</span>
<span class="k">override</span> <span class="k">val</span> <span class="n">config</span><span class="k">:</span> <span class="kt">F</span><span class="o">[</span><span class="kt">Config</span><span class="o">]</span> <span class="k">=</span> <span class="o">{</span>
<span class="cm">/**
* We can wrap it in an either, if we do that then we don't
* really need any other typeclasses, we can also say that
* we expect the result to be mapped to a monadError. That
* means that the F[_] used, should satisfy the MonadError
* constraint.
*/</span>
<span class="k">val</span> <span class="n">c</span> <span class="k">=</span> <span class="nc">ConfigSource</span><span class="o">.</span><span class="n">default</span>
<span class="o">.</span><span class="n">load</span><span class="o">[</span><span class="kt">Config</span><span class="o">]</span>
<span class="o">.</span><span class="n">left</span>
<span class="o">.</span><span class="n">map</span><span class="o">(</span><span class="n">pureconfig</span><span class="o">.</span><span class="n">error</span><span class="o">.</span><span class="nc">ConfigReaderException</span><span class="o">.</span><span class="n">apply</span><span class="o">)</span>
<span class="c1">// summon the monadError instance and convert to an either
</span> <span class="nc">MonadError</span><span class="o">[</span><span class="kt">F</span>, <span class="kt">Throwable</span><span class="o">].</span><span class="n">fromEither</span><span class="o">(</span><span class="n">c</span><span class="o">)</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>In the code above we want to communicate to the users that loading this config can fail. So we expect that the monad the user provides supports at least the functionality defined in <a href="https://typelevel.org/cats/api/cats/MonadError.html"><code class="highlighter-rouge">MonadError</code></a>. This could for instance be an <code class="highlighter-rouge">Either</code> or a <code class="highlighter-rouge">Try</code>. We define this dependency as a constraint on <code class="highlighter-rouge">F[_]</code> like this: <code class="highlighter-rouge">F[_]: MonadError[*[_], Throwable]</code>. This simpler syntax is provided by the kind project (https://github.com/typelevel/kind-projector), which allows for easier syntax. This is enabled in the <code class="highlighter-rouge">build.sbt</code>:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ThisBuild / scalaVersion := "2.13.4"
ThisBuild / version := "0.1.0-SNAPSHOT"
ThisBuild / organization := "org.smartjava"
ThisBuild / organizationName := "smartjava"
val CatsVersion = "3.1.1"
val Http4sVersion = "1.0.0-M23"
lazy val hello = (project in file("."))
.settings(
name := "Exploring Tagless",
libraryDependencies ++= Seq(
"org.typelevel" %% "cats-effect" % CatsVersion,
"com.github.pureconfig" %% "pureconfig" % "0.15.0",
"org.http4s" %% "http4s-blaze-server" % Http4sVersion,
"org.http4s" %% "http4s-dsl" % Http4sVersion,
"org.http4s" %% "http4s-blaze-client" % Http4sVersion,
"org.http4s" %% "http4s-circe" % Http4sVersion,
"ch.qos.logback" % "logback-core" % "1.2.3",
"org.slf4j" % "slf4j-api" % "1.7.30",
"ch.qos.logback" % "logback-classic" % "1.2.3",
"co.fs2" %% "fs2-core" % "3.0.4",
"co.fs2" %% "fs2-io" % "3.0.4",
"co.fs2" %% "fs2-reactive-streams" % "3.0.4",
// JSON Mapping
"io.circe" %% "circe-generic" % "0.12.3",
"io.circe" %% "circe-literal" % "0.12.3",
"org.mongodb.scala" %% "mongo-scala-driver" % "4.2.3"
),
scalacOptions += "-Ymacro-annotations"
)
addCompilerPlugin("org.typelevel" % "kind-projector" % "0.11.3" cross CrossVersion.full)
</code></pre></div></div>
<p>In the final main we’ll use this configuration something like this:</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">import</span> <span class="nn">cats.effect.IO</span>
<span class="k">implicit</span> <span class="k">val</span> <span class="n">configService</span> <span class="k">=</span> <span class="nc">LiveConfigInterpreter</span><span class="o">[</span><span class="kt">IO</span><span class="o">]()</span>
</code></pre></div></div>
<p>With the configuration out of the way we can look at configuring our HTTP4S client and server</p>
<h2 id="setting-up-http4s">Setting up HTTP4s</h2>
<p>We’ll start by looking at the client.</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">object</span> <span class="nc">http4sClient</span> <span class="o">{</span>
<span class="cm">/**
* Simple DSL to get an instance of a client
*/</span>
<span class="k">trait</span> <span class="nc">Http4sClientDSL</span><span class="o">[</span><span class="kt">F</span><span class="o">[</span><span class="k">_</span><span class="o">]]</span> <span class="o">{</span>
<span class="k">val</span> <span class="n">client</span><span class="k">:</span> <span class="kt">Resource</span><span class="o">[</span><span class="kt">F</span>, <span class="kt">Client</span><span class="o">[</span><span class="kt">F</span><span class="o">]]</span>
<span class="o">}</span>
<span class="k">object</span> <span class="nc">Http4sClientDSL</span> <span class="o">{</span>
<span class="k">def</span> <span class="n">apply</span><span class="o">[</span><span class="kt">F</span><span class="o">[</span><span class="k">_</span><span class="o">]](</span><span class="k">implicit</span> <span class="n">F</span><span class="k">:</span> <span class="kt">Http4sClientDSL</span><span class="o">[</span><span class="kt">F</span><span class="o">])</span><span class="k">:</span> <span class="kt">F.</span><span class="k">type</span> <span class="o">=</span> <span class="n">F</span>
<span class="o">}</span>
<span class="cm">/**
* Live version
*/</span>
<span class="k">case</span> <span class="k">class</span> <span class="nc">LiveHttp4sClientInterpreter</span><span class="o">[</span><span class="kt">F</span><span class="o">[</span><span class="k">_</span><span class="o">]</span><span class="kt">:</span> <span class="kt">Async</span><span class="o">]()(</span><span class="k">implicit</span> <span class="n">ec</span><span class="k">:</span> <span class="kt">ExecutionContext</span><span class="o">)</span> <span class="k">extends</span> <span class="nc">Http4sClientDSL</span><span class="o">[</span><span class="kt">F</span><span class="o">]</span> <span class="o">{</span>
<span class="k">override</span> <span class="k">val</span> <span class="n">client</span><span class="k">:</span> <span class="kt">Resource</span><span class="o">[</span><span class="kt">F</span>, <span class="kt">Client</span><span class="o">[</span><span class="kt">F</span><span class="o">]]</span> <span class="k">=</span> <span class="nc">BlazeClientBuilder</span><span class="o">[</span><span class="kt">F</span><span class="o">](</span><span class="n">ec</span><span class="o">).</span><span class="n">resource</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>If you look at the trait, we’re going to return a <code class="highlighter-rouge">Resource[F, Client[F]]</code>. Which basically allows us to get access to a <code class="highlighter-rouge">Client[F]</code> by calling <code class="highlighter-rouge">use</code> on the resource. Since HTTP4s already is based on Cats Effect, we only have to call <code class="highlighter-rouge">BlazeClientBuilder[F](ec).resource</code> and we’re done… well almost. If you look at the signature:</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="n">apply</span><span class="o">[</span><span class="kt">F</span><span class="o">[</span><span class="k">_</span><span class="o">]</span><span class="kt">:</span> <span class="kt">Async</span><span class="o">](</span><span class="n">executionContext</span><span class="k">:</span> <span class="kt">ExecutionContext</span><span class="o">)</span><span class="k">:</span> <span class="kt">BlazeClientBuilder</span><span class="o">[</span><span class="kt">F</span><span class="o">]</span> <span class="k">=</span>
</code></pre></div></div>
<p>This function requires an <code class="highlighter-rouge">Async[F]</code> to be in scope. So that’s why we also added it to the case class (which serves as our module / service). You can also see that this function requires an explicit execution context. It seems to be rather arbitrary when the <code class="highlighter-rouge">ExectionContext</code> is passed explicitly or implicitly, so in our module we expect this to be passed in implicitly.</p>
<p>The <code class="highlighter-rouge">Http4sServer</code> setup isn’t that different:</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">object</span> <span class="nc">http4sServer</span> <span class="o">{</span>
<span class="cm">/**
* This trait provides access to a Http4sServer dependency. It is
* wrapped in the F monad, so we can signal errors during setup.
*
* Wondering whether these should be called DSL, since these are
* just the low level dependencies.
*/</span>
<span class="k">trait</span> <span class="nc">Http4sServerDSL</span><span class="o">[</span><span class="kt">F</span><span class="o">[</span><span class="k">_</span><span class="o">]]</span> <span class="o">{</span>
<span class="k">val</span> <span class="n">server</span><span class="k">:</span> <span class="kt">Resource</span><span class="o">[</span><span class="kt">F</span>, <span class="kt">Server</span><span class="o">]</span>
<span class="o">}</span>
<span class="cm">/**
* Helper object to access the instance of the DSL that is in scope
*/</span>
<span class="k">object</span> <span class="nc">Http4ServerDSL</span> <span class="o">{</span>
<span class="k">def</span> <span class="n">apply</span><span class="o">[</span><span class="kt">F</span><span class="o">[</span><span class="k">_</span><span class="o">]](</span><span class="k">implicit</span> <span class="n">F</span><span class="k">:</span> <span class="kt">Http4sServerDSL</span><span class="o">[</span><span class="kt">F</span><span class="o">])</span><span class="k">:</span> <span class="kt">F.</span><span class="k">type</span> <span class="o">=</span> <span class="n">F</span>
<span class="o">}</span>
<span class="k">case</span> <span class="k">class</span> <span class="nc">routes</span><span class="o">[</span><span class="kt">F</span><span class="o">[</span><span class="k">_</span><span class="o">]</span><span class="kt">:</span> <span class="kt">Monad</span><span class="o">]()</span> <span class="o">{</span>
<span class="k">private</span> <span class="k">val</span> <span class="n">dsl</span> <span class="k">=</span> <span class="nc">Http4sDsl</span><span class="o">[</span><span class="kt">F</span><span class="o">]</span>
<span class="k">import</span> <span class="nn">dsl._</span>
<span class="k">val</span> <span class="n">routes</span><span class="k">:</span> <span class="kt">HttpRoutes</span><span class="o">[</span><span class="kt">F</span><span class="o">]</span> <span class="k">=</span> <span class="nc">HttpRoutes</span>
<span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">F</span><span class="o">]</span> <span class="o">{</span>
<span class="k">case</span> <span class="nc">GET</span> <span class="o">-></span> <span class="nc">Root</span> <span class="o">/</span> <span class="s">"users"</span> <span class="o">/</span> <span class="nc">IntVar</span><span class="o">(</span><span class="n">id</span><span class="o">)</span> <span class="k">=></span> <span class="o">{</span>
<span class="nc">Created</span><span class="o">(</span><span class="s">"A value here"</span><span class="o">)</span>
<span class="o">}</span>
<span class="k">case</span> <span class="nc">POST</span> <span class="o">-></span> <span class="nc">Root</span> <span class="o">/</span> <span class="s">"users"</span> <span class="k">=></span> <span class="o">{</span>
<span class="nc">Created</span><span class="o">(</span><span class="s">"And another one here"</span><span class="o">)</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="cm">/**
* Live instance of the HttpsServer service.
*
* @param ec
*/</span>
<span class="k">case</span> <span class="k">class</span> <span class="nc">LiveHttp4sServerInterpreter</span><span class="o">[</span>
<span class="kt">F</span><span class="o">[</span><span class="k">_</span><span class="o">]</span><span class="kt">:</span> <span class="kt">MonadCancel</span><span class="o">[</span><span class="kt">*</span><span class="o">[</span><span class="k">_</span><span class="o">]</span>, <span class="kt">Throwable</span><span class="o">]</span><span class="kt">:</span> <span class="kt">Async:</span> <span class="kt">ConfigDSL</span>
<span class="o">]()(</span><span class="k">implicit</span> <span class="n">ec</span><span class="k">:</span> <span class="kt">ExecutionContext</span><span class="o">)</span>
<span class="k">extends</span> <span class="nc">Http4sServerDSL</span><span class="o">[</span><span class="kt">F</span><span class="o">]</span> <span class="o">{</span>
<span class="k">override</span> <span class="k">val</span> <span class="n">server</span><span class="k">:</span> <span class="kt">Resource</span><span class="o">[</span><span class="kt">F</span>, <span class="kt">Server</span><span class="o">]</span> <span class="k">=</span>
<span class="k">for</span> <span class="o">{</span>
<span class="n">config</span> <span class="k"><-</span> <span class="nc">Resource</span><span class="o">.</span><span class="n">eval</span><span class="o">(</span><span class="nc">ConfigDSL</span><span class="o">[</span><span class="kt">F</span><span class="o">].</span><span class="n">config</span><span class="o">)</span>
<span class="n">server</span> <span class="k"><-</span> <span class="nc">BlazeServerBuilder</span><span class="o">[</span><span class="kt">F</span><span class="o">](</span><span class="n">ec</span><span class="o">)</span>
<span class="o">.</span><span class="n">bindHttp</span><span class="o">(</span><span class="n">config</span><span class="o">.</span><span class="n">apiConfig</span><span class="o">.</span><span class="n">port</span><span class="o">,</span> <span class="n">config</span><span class="o">.</span><span class="n">apiConfig</span><span class="o">.</span><span class="n">endpoint</span><span class="o">)</span>
<span class="o">.</span><span class="n">withHttpApp</span><span class="o">(</span><span class="n">routes</span><span class="o">[</span><span class="kt">F</span><span class="o">]().</span><span class="n">routes</span><span class="o">.</span><span class="n">orNotFound</span><span class="o">)</span>
<span class="o">.</span><span class="n">resource</span>
<span class="o">}</span> <span class="k">yield</span> <span class="o">(</span>
<span class="n">server</span>
<span class="o">)</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>As you can see in the code above, the process is the same. We pass in a couple of effects from Cats Effect (<code class="highlighter-rouge">MonadCancel</code> and <code class="highlighter-rouge">Async</code>) since those are required by <code class="highlighter-rouge">BlazeServerBuilder[F]</code>. Additionally we also add <code class="highlighter-rouge">ConfigDSL</code> to the list of constraints. This means that to create this module, we need to make sure a <code class="highlighter-rouge">ConfigDSL[F]</code> is in scope as well. If we want to instantiate this module we’d need to do something like this:</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">implicit</span> <span class="k">val</span> <span class="n">ec</span> <span class="k">=</span> <span class="nc">ExecutionContext</span><span class="o">.</span><span class="n">global</span>
<span class="k">implicit</span> <span class="k">val</span> <span class="n">configService</span> <span class="k">=</span> <span class="nc">LiveConfigInterpreter</span><span class="o">[</span><span class="kt">IO</span><span class="o">]()</span>
<span class="c1">// implicit since it'll be used a dependency later on.
</span><span class="k">implicit</span> <span class="k">val</span> <span class="n">http4sClient</span> <span class="k">=</span> <span class="n">services</span><span class="o">.</span><span class="n">http4sClient</span><span class="o">.</span><span class="nc">LiveHttp4sClientInterpreter</span><span class="o">[</span><span class="kt">IO</span><span class="o">]()</span>
<span class="k">val</span> <span class="n">http4s</span> <span class="k">=</span> <span class="nc">LiveHttp4sServerInterpreter</span><span class="o">[</span><span class="kt">IO</span><span class="o">]()</span>
</code></pre></div></div>
<p>That way we can easily get access to the Http4S server.</p>
<h2 id="configuring-mongodb-for-data-storage">Configuring MongoDB for data storage</h2>
<p>Now let’s look at the bottom layer of the application where we want to store data in a mongo database. First we’ll look at how to acquire the mongo client:</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">object</span> <span class="nc">mongo</span> <span class="o">{</span>
<span class="k">trait</span> <span class="nc">MongoDBConnectionDSL</span><span class="o">[</span><span class="kt">F</span><span class="o">[</span><span class="k">_</span><span class="o">]]</span> <span class="o">{</span>
<span class="k">val</span> <span class="n">mongoClient</span><span class="k">:</span> <span class="kt">Resource</span><span class="o">[</span><span class="kt">F</span>, <span class="kt">MongoClient</span><span class="o">]</span>
<span class="o">}</span>
<span class="k">object</span> <span class="nc">MongoDBConnectionDSL</span> <span class="o">{</span>
<span class="k">def</span> <span class="n">apply</span><span class="o">[</span><span class="kt">F</span><span class="o">[</span><span class="k">_</span><span class="o">]](</span><span class="k">implicit</span> <span class="n">F</span><span class="k">:</span> <span class="kt">MongoDBConnectionDSL</span><span class="o">[</span><span class="kt">F</span><span class="o">])</span><span class="k">:</span> <span class="kt">F.</span><span class="k">type</span> <span class="o">=</span> <span class="n">F</span>
<span class="o">}</span>
<span class="k">case</span> <span class="k">class</span> <span class="nc">LiveMongoDBConnectionInterpreter</span><span class="o">[</span><span class="kt">F</span><span class="o">[</span><span class="k">_</span><span class="o">]</span><span class="kt">:</span> <span class="kt">Sync:</span> <span class="kt">ConfigDSL</span><span class="o">]()</span> <span class="k">extends</span> <span class="nc">MongoDBConnectionDSL</span><span class="o">[</span><span class="kt">F</span><span class="o">]</span> <span class="o">{</span>
<span class="k">val</span> <span class="nc">ME</span> <span class="k">=</span> <span class="nc">MonadError</span><span class="o">[</span><span class="kt">F</span>, <span class="kt">Throwable</span><span class="o">]</span>
<span class="k">val</span> <span class="nc">Config</span> <span class="k">=</span> <span class="nc">ConfigDSL</span><span class="o">[</span><span class="kt">F</span><span class="o">]</span>
<span class="k">override</span> <span class="k">val</span> <span class="n">mongoClient</span> <span class="k">=</span> <span class="nc">Resource</span><span class="o">.</span><span class="n">make</span><span class="o">(</span><span class="n">acquireConnection</span><span class="o">)(</span><span class="n">releaseConnection</span><span class="o">)</span>
<span class="cm">/**
* Acquire a connection
*/</span>
<span class="k">private</span> <span class="k">def</span> <span class="n">acquireConnection</span><span class="o">()(</span><span class="k">implicit</span> <span class="n">config</span><span class="k">:</span> <span class="kt">ConfigDSL</span><span class="o">[</span><span class="kt">F</span><span class="o">])</span> <span class="k">=</span>
<span class="k">for</span> <span class="o">{</span>
<span class="n">c</span> <span class="k"><-</span> <span class="n">config</span><span class="o">.</span><span class="n">config</span>
<span class="n">r</span> <span class="k"><-</span> <span class="nc">Sync</span><span class="o">[</span><span class="kt">F</span><span class="o">].</span><span class="n">blocking</span><span class="o">(</span><span class="nc">MongoClient</span><span class="o">(</span><span class="n">c</span><span class="o">.</span><span class="n">dbConfig</span><span class="o">.</span><span class="n">endpoint</span><span class="o">))</span>
<span class="o">}</span> <span class="k">yield</span> <span class="o">{</span>
<span class="n">r</span>
<span class="o">}</span>
<span class="cm">/**
* Release the mongo client, wrap in sync since it's effectful
*/</span>
<span class="k">private</span> <span class="k">def</span> <span class="n">releaseConnection</span><span class="o">(</span><span class="n">mongoClient</span><span class="k">:</span> <span class="kt">MongoClient</span><span class="o">)</span> <span class="k">=</span> <span class="nc">Sync</span><span class="o">[</span><span class="kt">F</span><span class="o">].</span><span class="n">blocking</span><span class="o">(</span><span class="n">mongoClient</span><span class="o">.</span><span class="n">close</span><span class="o">)</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Here you can again see a very simple trait explaining what this DSL offers (not much in this simple example), and an implementation (still agnostic on <code class="highlighter-rouge">F[_]</code>). The only implementation here is that we require a <code class="highlighter-rouge">Sync[F]</code> implementation, which we use to call the effectfull instantiation of the <code class="highlighter-rouge">MongoClient</code>. Since we do this as a resource we have to provide a function to acquire the resource <code class="highlighter-rouge">acquireConnection</code> and a function to release the resource again <code class="highlighter-rouge">releaseConnection</code>. Using <code class="highlighter-rouge">Sync[F]</code> will defer the code within the <code class="highlighter-rouge">blocking</code> function until the effect is evaluated.</p>
<p>Now that we’ve got a way to access the mongo client, we can also implement storing and retrieving our domain entity (a <code class="highlighter-rouge">Temperature</code> object). The code below is almost exactly the same as it was in the ZIO example with <code class="highlighter-rouge">ZStream</code>, so we won’t explain the details:</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">object</span> <span class="nc">storage</span> <span class="o">{</span>
<span class="k">trait</span> <span class="nc">TemperatureStorageDSL</span><span class="o">[</span><span class="kt">F</span><span class="o">[</span><span class="k">_</span><span class="o">]]</span> <span class="o">{</span>
<span class="k">def</span> <span class="n">insert</span><span class="o">(</span><span class="n">temperature</span><span class="k">:</span> <span class="kt">Temperature</span><span class="o">)</span><span class="k">:</span> <span class="kt">F</span><span class="o">[</span><span class="kt">Unit</span><span class="o">]</span>
<span class="k">def</span> <span class="n">getAll</span><span class="o">()</span><span class="k">:</span> <span class="kt">F</span><span class="o">[</span><span class="kt">List</span><span class="o">[</span><span class="kt">Temperature</span><span class="o">]]</span>
<span class="o">}</span>
<span class="k">object</span> <span class="nc">TemperatureStorageDSL</span> <span class="o">{</span>
<span class="k">def</span> <span class="n">apply</span><span class="o">[</span><span class="kt">F</span><span class="o">[</span><span class="k">_</span><span class="o">]](</span><span class="k">implicit</span> <span class="n">F</span><span class="k">:</span> <span class="kt">TemperatureStorageDSL</span><span class="o">[</span><span class="kt">F</span><span class="o">])</span><span class="k">:</span> <span class="kt">F.</span><span class="k">type</span> <span class="o">=</span> <span class="n">F</span>
<span class="o">}</span>
<span class="k">case</span> <span class="k">class</span> <span class="nc">TemperatureStorageInterpreter</span><span class="o">[</span>
<span class="kt">F</span><span class="o">[</span><span class="k">_</span><span class="o">]</span><span class="kt">:</span> <span class="kt">MonadCancel</span><span class="o">[</span><span class="kt">*</span><span class="o">[</span><span class="k">_</span><span class="o">]</span>, <span class="kt">Throwable</span><span class="o">]</span><span class="kt">:</span> <span class="kt">Async:</span> <span class="kt">ConfigDSL:</span> <span class="kt">MongoDBConnectionDSL</span>
<span class="o">]()</span> <span class="k">extends</span> <span class="nc">TemperatureStorageDSL</span><span class="o">[</span><span class="kt">F</span><span class="o">]</span> <span class="o">{</span>
<span class="k">val</span> <span class="n">temperatureCodecProvider</span> <span class="k">=</span> <span class="nc">Macros</span><span class="o">.</span><span class="n">createCodecProvider</span><span class="o">[</span><span class="kt">Temperature</span><span class="o">]()</span>
<span class="k">val</span> <span class="n">codecRegistry</span> <span class="k">=</span> <span class="n">fromRegistries</span><span class="o">(</span><span class="n">fromProviders</span><span class="o">(</span><span class="n">temperatureCodecProvider</span><span class="o">),</span> <span class="nc">DEFAULT_CODEC_REGISTRY</span><span class="o">)</span>
<span class="k">val</span> <span class="nc">ME</span> <span class="k">=</span> <span class="nc">MonadError</span><span class="o">[</span><span class="kt">F</span>, <span class="kt">Throwable</span><span class="o">]</span>
<span class="k">override</span> <span class="k">def</span> <span class="n">insert</span><span class="o">(</span><span class="n">temperature</span><span class="k">:</span> <span class="kt">Temperature</span><span class="o">)</span><span class="k">:</span> <span class="kt">F</span><span class="o">[</span><span class="kt">Unit</span><span class="o">]</span> <span class="k">=</span>
<span class="n">withCollection</span><span class="o">[</span><span class="kt">Unit</span>, <span class="kt">Temperature</span><span class="o">]</span> <span class="o">{</span> <span class="n">c</span> <span class="k">=></span>
<span class="n">c</span><span class="o">.</span><span class="n">insertOne</span><span class="o">(</span><span class="n">temperature</span><span class="o">)</span>
<span class="o">.</span><span class="n">toStream</span><span class="o">[</span><span class="kt">F</span><span class="o">]</span>
<span class="o">.</span><span class="n">compile</span>
<span class="o">.</span><span class="n">toList</span>
<span class="o">.</span><span class="n">map</span><span class="o">(</span><span class="k">_</span><span class="o">.</span><span class="n">length</span><span class="o">)</span>
<span class="c1">// naively assume that when we get 1 insert result everything is fine
</span> <span class="o">.</span><span class="n">flatMap</span> <span class="o">{</span>
<span class="k">case</span> <span class="mi">1</span> <span class="k">=></span> <span class="nc">ME</span><span class="o">.</span><span class="n">pure</span><span class="o">()</span>
<span class="k">case</span> <span class="k">_</span> <span class="k">=></span> <span class="nc">ME</span><span class="o">.</span><span class="n">raiseError</span><span class="o">[</span><span class="kt">Unit</span><span class="o">](</span><span class="k">new</span> <span class="nc">IllegalArgumentException</span><span class="o">(</span><span class="s">"Expected result from mongodb"</span><span class="o">))</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="k">override</span> <span class="k">def</span> <span class="n">getAll</span><span class="o">()</span><span class="k">:</span> <span class="kt">F</span><span class="o">[</span><span class="kt">List</span><span class="o">[</span><span class="kt">Temperature</span><span class="o">]]</span> <span class="k">=</span> <span class="o">{</span>
<span class="n">withCollection</span><span class="o">[</span><span class="kt">List</span><span class="o">[</span><span class="kt">Temperature</span><span class="o">]</span>, <span class="kt">Temperature</span><span class="o">]</span> <span class="o">{</span>
<span class="k">_</span><span class="o">.</span><span class="n">find</span><span class="o">()</span>
<span class="o">.</span><span class="n">toStream</span><span class="o">[</span><span class="kt">F</span><span class="o">]</span>
<span class="o">.</span><span class="n">compile</span>
<span class="o">.</span><span class="n">toList</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="cm">/**
* Get the database and collection to which to store.
*
* @param f function to call within the context of this collection
* @return result of wrapped
*/</span>
<span class="k">private</span> <span class="k">def</span> <span class="n">withCollection</span><span class="o">[</span><span class="kt">A</span>, <span class="kt">T:</span> <span class="kt">ClassTag</span><span class="o">](</span>
<span class="n">f</span><span class="k">:</span> <span class="kt">MongoCollection</span><span class="o">[</span><span class="kt">T</span><span class="o">]</span> <span class="k">=></span> <span class="n">F</span><span class="o">[</span><span class="kt">A</span><span class="o">]</span>
<span class="o">)</span><span class="k">:</span> <span class="kt">F</span><span class="o">[</span><span class="kt">A</span><span class="o">]</span> <span class="k">=</span> <span class="nc">MongoDBConnectionDSL</span><span class="o">[</span><span class="kt">F</span><span class="o">].</span><span class="n">mongoClient</span><span class="o">.</span><span class="n">use</span><span class="o">[</span><span class="kt">A</span><span class="o">]</span> <span class="o">{</span> <span class="n">mongoClient</span> <span class="k">=></span>
<span class="k">val</span> <span class="n">collection</span> <span class="k">=</span>
<span class="n">mongoClient</span>
<span class="o">.</span><span class="n">getDatabase</span><span class="o">(</span><span class="s">"sampleservice"</span><span class="o">)</span>
<span class="o">.</span><span class="n">withCodecRegistry</span><span class="o">(</span><span class="n">codecRegistry</span><span class="o">)</span>
<span class="o">.</span><span class="n">getCollection</span><span class="o">[</span><span class="kt">T</span><span class="o">](</span><span class="s">"temperatures"</span><span class="o">)</span>
<span class="n">f</span><span class="o">(</span><span class="n">collection</span><span class="o">)</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>The only interesting part here are the dependencies we want for this module:</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">case</span> <span class="k">class</span> <span class="nc">TemperatureStorageInterpreter</span><span class="o">[</span>
<span class="kt">F</span><span class="o">[</span><span class="k">_</span><span class="o">]</span><span class="kt">:</span> <span class="kt">MonadCancel</span><span class="o">[</span><span class="kt">*</span><span class="o">[</span><span class="k">_</span><span class="o">]</span>, <span class="kt">Throwable</span><span class="o">]</span><span class="kt">:</span> <span class="kt">Async:</span> <span class="kt">ConfigDSL:</span> <span class="kt">MongoDBConnectionDSL</span>
<span class="o">]()</span> <span class="k">extends</span> <span class="nc">TemperatureStorageDSL</span><span class="o">[</span><span class="kt">F</span><span class="o">]</span> <span class="o">{</span>
</code></pre></div></div>
<p>So what we need are a couple of classes from Cats Effect, we require an instance of the <code class="highlighter-rouge">ConfigDSL</code> and we require an instance of the <code class="highlighter-rouge">MongoDBConnectionDSL</code> as well.</p>
<p>And of course since the client is a resource we have to use <code class="highlighter-rouge">use</code> to access the client in a safe manner:</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">private</span> <span class="k">def</span> <span class="n">withCollection</span><span class="o">[</span><span class="kt">A</span>, <span class="kt">T:</span> <span class="kt">ClassTag</span><span class="o">](</span>
<span class="n">f</span><span class="k">:</span> <span class="kt">MongoCollection</span><span class="o">[</span><span class="kt">T</span><span class="o">]</span> <span class="k">=></span> <span class="n">F</span><span class="o">[</span><span class="kt">A</span><span class="o">]</span>
<span class="o">)</span><span class="k">:</span> <span class="kt">F</span><span class="o">[</span><span class="kt">A</span><span class="o">]</span> <span class="k">=</span> <span class="nc">MongoDBConnectionDSL</span><span class="o">[</span><span class="kt">F</span><span class="o">].</span><span class="n">mongoClient</span><span class="o">.</span><span class="n">use</span><span class="o">[</span><span class="kt">A</span><span class="o">]</span> <span class="o">{</span> <span class="n">mongoClient</span> <span class="k">=></span>
<span class="k">val</span> <span class="n">collection</span> <span class="k">=</span>
<span class="n">mongoClient</span>
<span class="o">.</span><span class="n">getDatabase</span><span class="o">(</span><span class="s">"sampleservice"</span><span class="o">)</span>
<span class="o">.</span><span class="n">withCodecRegistry</span><span class="o">(</span><span class="n">codecRegistry</span><span class="o">)</span>
<span class="o">.</span><span class="n">getCollection</span><span class="o">[</span><span class="kt">T</span><span class="o">](</span><span class="s">"temperatures"</span><span class="o">)</span>
<span class="n">f</span><span class="o">(</span><span class="n">collection</span><span class="o">)</span>
<span class="o">}</span>
</code></pre></div></div>
<p>When this effect is evaluated, the <code class="highlighter-rouge">acquire</code> function we defined for retrieving a <code class="highlighter-rouge">MongoClient</code> will be called, and at the end of the function the <code class="highlighter-rouge">release</code> function we defined is called.</p>
<p>And the final part that is interesting is the integration we have here with FS2. The mongo client we use is based on the <a href="https://www.reactive-streams.org/">Reactive Streams</a> implementation. Luckily we can just integrate with that directly with FS2:</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="n">c</span><span class="o">.</span><span class="n">insertOne</span><span class="o">(</span><span class="n">temperature</span><span class="o">)</span>
<span class="o">.</span><span class="n">toStream</span><span class="o">[</span><span class="kt">F</span><span class="o">]</span>
</code></pre></div></div>
<p>That will result in a monad agnostic stream (<code class="highlighter-rouge">def toStream[F[_]: Async]: Stream[F, A]</code>) and the only thing we need to do is make sure the monad we provide has the <code class="highlighter-rouge">Async</code> constraint.</p>
<h2 id="create-a-stream-of-temperature-updates">Create a stream of temperature updates</h2>
<p>The final thing we need to do is create a client that polls an http endpoint, and returns a stream of temperature updates. For that we just define a new module where we expose a single value which is a stream of temperatures. The type is defined like this: <code class="highlighter-rouge">F[Stream[F, Temperature]]</code>. This means that retrieving the stream itself is also effectful (since we use information from the configuration) and when we evaluate the stream we also do this within the context of the supplied <code class="highlighter-rouge">F[_]</code>. So our stream itself uses functions which aren’t pure (e.g simply calculations), in this case we make a call using the passed in <code class="highlighter-rouge">Http4sClientDSL</code>:</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">object</span> <span class="nc">tempClient</span> <span class="o">{</span>
<span class="k">trait</span> <span class="nc">TempClientDSL</span><span class="o">[</span><span class="kt">F</span><span class="o">[</span><span class="k">_</span><span class="o">]]</span> <span class="o">{</span>
<span class="k">val</span> <span class="n">temperatureStream</span><span class="k">:</span> <span class="kt">F</span><span class="o">[</span><span class="kt">Stream</span><span class="o">[</span><span class="kt">F</span>, <span class="kt">Temperature</span><span class="o">]]</span>
<span class="o">}</span>
<span class="k">case</span> <span class="k">class</span> <span class="nc">LiveTempClientInterpreter</span><span class="o">[</span><span class="kt">F</span><span class="o">[</span><span class="k">_</span><span class="o">]</span><span class="kt">:</span> <span class="kt">MonadError</span><span class="o">[</span><span class="kt">*</span><span class="o">[</span><span class="k">_</span><span class="o">]</span>, <span class="kt">Throwable</span><span class="o">]</span><span class="kt">:</span> <span class="kt">Temporal:</span> <span class="kt">ConfigDSL:</span> <span class="kt">Http4sClientDSL</span><span class="o">]()</span>
<span class="k">extends</span> <span class="nc">TempClientDSL</span><span class="o">[</span><span class="kt">F</span><span class="o">]</span> <span class="o">{</span>
<span class="k">import</span> <span class="nn">org.http4s.circe._</span>
<span class="k">import</span> <span class="nn">io.circe.generic.auto._</span>
<span class="k">import</span> <span class="nn">TempClientLive.OpenWeather._</span>
<span class="k">implicit</span> <span class="k">val</span> <span class="n">userDecoder</span> <span class="k">=</span> <span class="n">jsonOf</span><span class="o">[</span><span class="kt">F</span>, <span class="kt">OWResult</span><span class="o">]</span>
<span class="cm">/**
* Get a stream in an effectful way.
*/</span>
<span class="k">override</span> <span class="k">val</span> <span class="n">temperatureStream</span><span class="k">:</span> <span class="kt">F</span><span class="o">[</span><span class="kt">Stream</span><span class="o">[</span><span class="kt">F</span>, <span class="kt">Temperature</span><span class="o">]]</span> <span class="k">=</span> <span class="k">for</span> <span class="o">{</span>
<span class="n">config</span> <span class="k"><-</span> <span class="nc">ConfigDSL</span><span class="o">[</span><span class="kt">F</span><span class="o">].</span><span class="n">config</span>
<span class="n">stream</span> <span class="k">=</span> <span class="o">(</span><span class="nc">Stream</span><span class="o">(</span><span class="mi">1</span><span class="o">)</span> <span class="o">++</span> <span class="nc">Stream</span><span class="o">.</span><span class="n">awakeEvery</span><span class="o">(</span><span class="n">config</span><span class="o">.</span><span class="n">temperatureConfig</span><span class="o">.</span><span class="n">interval</span><span class="o">)).</span><span class="n">evalMap</span> <span class="o">{</span> <span class="k">_</span> <span class="k">=></span>
<span class="n">makeTemperatureCall</span><span class="o">(</span><span class="n">config</span><span class="o">.</span><span class="n">temperatureConfig</span><span class="o">.</span><span class="n">endpoint</span> <span class="o">+</span> <span class="n">config</span><span class="o">.</span><span class="n">temperatureConfig</span><span class="o">.</span><span class="n">apiKey</span><span class="o">)</span>
<span class="o">}</span>
<span class="o">}</span> <span class="k">yield</span> <span class="o">{</span>
<span class="n">stream</span>
<span class="o">}</span>
<span class="cm">/**
* Make an actual call to a rest endpoint
*/</span>
<span class="k">private</span> <span class="k">def</span> <span class="n">makeTemperatureCall</span><span class="o">(</span><span class="n">url</span><span class="k">:</span> <span class="kt">String</span><span class="o">)(</span><span class="k">implicit</span> <span class="n">clientDSL</span><span class="k">:</span> <span class="kt">Http4sClientDSL</span><span class="o">[</span><span class="kt">F</span><span class="o">])</span><span class="k">:</span> <span class="kt">F</span><span class="o">[</span><span class="kt">Temperature</span><span class="o">]</span> <span class="k">=</span>
<span class="n">clientDSL</span><span class="o">.</span><span class="n">client</span><span class="o">.</span><span class="n">use</span> <span class="o">{</span>
<span class="k">_</span><span class="o">.</span><span class="n">expect</span><span class="o">[</span><span class="kt">TempClientLive.OpenWeather.OWResult</span><span class="o">](</span><span class="n">url</span><span class="o">).</span><span class="n">map</span><span class="o">(</span><span class="n">t</span> <span class="k">=></span> <span class="nc">Temperature</span><span class="o">(</span><span class="n">t</span><span class="o">.</span><span class="n">dt</span><span class="o">,</span> <span class="n">t</span><span class="o">.</span><span class="n">main</span><span class="o">.</span><span class="n">temp</span><span class="o">))</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="k">object</span> <span class="nc">TempClientLive</span> <span class="o">{</span>
<span class="k">object</span> <span class="nc">OpenWeather</span> <span class="o">{</span>
<span class="c1">// the openweather model
</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">OWResult</span><span class="o">(</span><span class="n">coord</span><span class="k">:</span> <span class="kt">OWCoord</span><span class="o">,</span> <span class="n">main</span><span class="k">:</span> <span class="kt">OWMain</span><span class="o">,</span> <span class="n">visibility</span><span class="k">:</span> <span class="kt">Integer</span><span class="o">,</span> <span class="n">wind</span><span class="k">:</span> <span class="kt">OWWind</span><span class="o">,</span> <span class="n">dt</span><span class="k">:</span> <span class="kt">Long</span><span class="o">)</span>
<span class="k">case</span> <span class="k">class</span> <span class="nc">OWCoord</span><span class="o">(</span><span class="n">lat</span><span class="k">:</span> <span class="kt">Double</span><span class="o">,</span> <span class="n">lon</span><span class="k">:</span> <span class="kt">Double</span><span class="o">)</span>
<span class="k">case</span> <span class="k">class</span> <span class="nc">OWMain</span><span class="o">(</span>
<span class="n">temp</span><span class="k">:</span> <span class="kt">Double</span><span class="o">,</span>
<span class="n">feels_like</span><span class="k">:</span> <span class="kt">Double</span><span class="o">,</span>
<span class="n">temp_min</span><span class="k">:</span> <span class="kt">Double</span><span class="o">,</span>
<span class="n">temp_max</span><span class="k">:</span> <span class="kt">Double</span><span class="o">,</span>
<span class="n">pressure</span><span class="k">:</span> <span class="kt">Int</span><span class="o">,</span>
<span class="n">humidity</span><span class="k">:</span> <span class="kt">Int</span>
<span class="o">)</span>
<span class="k">case</span> <span class="k">class</span> <span class="nc">OWWind</span><span class="o">(</span><span class="n">speed</span><span class="k">:</span> <span class="kt">Double</span><span class="o">,</span> <span class="n">deg</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span> <span class="n">gust</span><span class="k">:</span> <span class="kt">Double</span><span class="o">)</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>When looking at this implementation you can see that in the case class definition we define a number of dependencies, just like we did for the other modules. The interesting part here is where we actually make and use the FS2 stream:</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="cm">/**
* Get a stream in an effectful way.
*/</span>
<span class="k">override</span> <span class="k">val</span> <span class="n">temperatureStream</span><span class="k">:</span> <span class="kt">F</span><span class="o">[</span><span class="kt">Stream</span><span class="o">[</span><span class="kt">F</span>, <span class="kt">Temperature</span><span class="o">]]</span> <span class="k">=</span> <span class="k">for</span> <span class="o">{</span>
<span class="n">config</span> <span class="k"><-</span> <span class="nc">ConfigDSL</span><span class="o">[</span><span class="kt">F</span><span class="o">].</span><span class="n">config</span>
<span class="n">stream</span> <span class="k">=</span> <span class="o">(</span><span class="nc">Stream</span><span class="o">(</span><span class="mi">1</span><span class="o">)</span> <span class="o">++</span> <span class="nc">Stream</span><span class="o">.</span><span class="n">awakeEvery</span><span class="o">(</span><span class="n">config</span><span class="o">.</span><span class="n">temperatureConfig</span><span class="o">.</span><span class="n">interval</span><span class="o">)).</span><span class="n">evalMap</span> <span class="o">{</span> <span class="k">_</span> <span class="k">=></span>
<span class="n">makeTemperatureCall</span><span class="o">(</span><span class="n">config</span><span class="o">.</span><span class="n">temperatureConfig</span><span class="o">.</span><span class="n">endpoint</span> <span class="o">+</span> <span class="n">config</span><span class="o">.</span><span class="n">temperatureConfig</span><span class="o">.</span><span class="n">apiKey</span><span class="o">)</span>
<span class="o">}</span>
<span class="o">}</span> <span class="k">yield</span> <span class="o">{</span>
<span class="n">stream</span>
<span class="o">}</span>
<span class="cm">/**
* Make an actual call to a rest endpoint
*/</span>
<span class="k">private</span> <span class="k">def</span> <span class="n">makeTemperatureCall</span><span class="o">(</span><span class="n">url</span><span class="k">:</span> <span class="kt">String</span><span class="o">)(</span><span class="k">implicit</span> <span class="n">clientDSL</span><span class="k">:</span> <span class="kt">Http4sClientDSL</span><span class="o">[</span><span class="kt">F</span><span class="o">])</span><span class="k">:</span> <span class="kt">F</span><span class="o">[</span><span class="kt">Temperature</span><span class="o">]</span> <span class="k">=</span>
<span class="n">clientDSL</span><span class="o">.</span><span class="n">client</span><span class="o">.</span><span class="n">use</span> <span class="o">{</span>
<span class="k">_</span><span class="o">.</span><span class="n">expect</span><span class="o">[</span><span class="kt">TempClientLive.OpenWeather.OWResult</span><span class="o">](</span><span class="n">url</span><span class="o">).</span><span class="n">map</span><span class="o">(</span><span class="n">t</span> <span class="k">=></span> <span class="nc">Temperature</span><span class="o">(</span><span class="n">t</span><span class="o">.</span><span class="n">dt</span><span class="o">,</span> <span class="n">t</span><span class="o">.</span><span class="n">main</span><span class="o">.</span><span class="n">temp</span><span class="o">))</span>
<span class="o">}</span>
</code></pre></div></div>
<p>We concatenate two stream together, this means that as soon as we evaluate this effect our stream immediately emits a value, and then it’ll emit values at the specified interval. Whenever a value is emitted, we run an effectfull function using <code class="highlighter-rouge">evalMap</code> which uses the <code class="highlighter-rouge">Http4sClientDSL[F]</code> to make the actual call.</p>
<p>The interesting thing is, that all this is still done without knowing anything (well almost anything) about the <code class="highlighter-rouge">F[_]</code> monad.</p>
<h2 id="glue-everything-together">Glue everything together</h2>
<p>All that is left to do is tie everything together and select the IO monad that we want to use:</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">object</span> <span class="nc">Runner</span> <span class="k">extends</span> <span class="nc">IOApp</span> <span class="o">{</span>
<span class="k">def</span> <span class="n">run</span><span class="o">(</span><span class="n">args</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">String</span><span class="o">])</span><span class="k">:</span> <span class="kt">IO</span><span class="o">[</span><span class="kt">ExitCode</span><span class="o">]</span> <span class="k">=</span> <span class="o">{</span>
<span class="c1">// get all the services, and add them to the implicit scope
</span> <span class="k">implicit</span> <span class="k">val</span> <span class="n">ec</span> <span class="k">=</span> <span class="nc">ExecutionContext</span><span class="o">.</span><span class="n">global</span>
<span class="c1">// the concrete implementations mapped to the IO monad
</span> <span class="k">implicit</span> <span class="k">val</span> <span class="n">configService</span> <span class="k">=</span> <span class="nc">LiveConfigInterpreter</span><span class="o">[</span><span class="kt">IO</span><span class="o">]()</span>
<span class="k">implicit</span> <span class="k">val</span> <span class="n">http4sClient</span> <span class="k">=</span> <span class="n">services</span><span class="o">.</span><span class="n">http4sClient</span><span class="o">.</span><span class="nc">LiveHttp4sClientInterpreter</span><span class="o">[</span><span class="kt">IO</span><span class="o">]()</span>
<span class="k">implicit</span> <span class="k">val</span> <span class="nc">MongoDBConnectionDSL</span> <span class="k">=</span> <span class="n">mongo</span><span class="o">.</span><span class="nc">LiveMongoDBConnectionInterpreter</span><span class="o">[</span><span class="kt">IO</span><span class="o">]()</span>
<span class="k">val</span> <span class="n">http4s</span> <span class="k">=</span> <span class="nc">LiveHttp4sServerInterpreter</span><span class="o">[</span><span class="kt">IO</span><span class="o">]()</span>
<span class="k">implicit</span> <span class="k">val</span> <span class="n">tempClient</span> <span class="k">=</span> <span class="n">services</span><span class="o">.</span><span class="n">tempClient</span><span class="o">.</span><span class="nc">LiveTempClientInterpreter</span><span class="o">[</span><span class="kt">IO</span><span class="o">]()</span>
<span class="k">val</span> <span class="n">tempStorage</span> <span class="k">=</span> <span class="n">storage</span><span class="o">.</span><span class="nc">TemperatureStorageInterpreter</span><span class="o">[</span><span class="kt">IO</span><span class="o">]()</span>
<span class="k">val</span> <span class="n">program</span> <span class="k">=</span> <span class="k">for</span> <span class="o">{</span>
<span class="c1">// consume the stream
</span> <span class="n">stream</span> <span class="k"><-</span> <span class="n">tempClient</span><span class="o">.</span><span class="n">temperatureStream</span>
<span class="k">_</span> <span class="k"><-</span> <span class="n">stream</span>
<span class="o">.</span><span class="n">evalMap</span><span class="o">(</span><span class="n">tempStorage</span><span class="o">.</span><span class="n">insert</span><span class="o">)</span>
<span class="o">.</span><span class="n">compile</span>
<span class="o">.</span><span class="n">drain</span>
<span class="o">.</span><span class="n">start</span>
<span class="c1">// start the server and keep running
</span> <span class="n">server</span> <span class="k"><-</span> <span class="n">http4s</span><span class="o">.</span><span class="n">server</span><span class="o">.</span><span class="n">use</span><span class="o">(</span><span class="k">_</span> <span class="k">=></span> <span class="nc">IO</span><span class="o">.</span><span class="n">never</span><span class="o">).</span><span class="n">as</span><span class="o">(</span><span class="nc">ExitCode</span><span class="o">.</span><span class="nc">Success</span><span class="o">)</span>
<span class="o">}</span> <span class="k">yield</span> <span class="o">{</span>
<span class="n">server</span>
<span class="o">}</span>
<span class="c1">// return the program to execute
</span> <span class="n">program</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>The program above uses the Cats IO monad as the monad to run everything in. What you see here is that the dependencies that are injected are defined as <code class="highlighter-rouge">implicit val</code> so they are automatically applied when we create the other modules. The program itself is just a simple for-comprehension which:</p>
<ol>
<li>Gets the stream of temperatures, and drains them. Whenever we receive a new temperature we store it in the database.</li>
<li>We use <code class="highlighter-rouge">.start</code> to start the stream, since that allows us to start this effect in the background:</li>
</ol>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> * Start execution of the source suspended in the `IO` context.
*
* This can be used for non-deterministic / concurrent execution.
</code></pre></div></div>
<ol>
<li>Then we start the webserver as well, which we’ll just keep running forever.</li>
</ol>
<p>The result is a <code class="highlighter-rouge">val program: IO[ExitCode]</code>, which is also the expected type of the <code class="highlighter-rouge">def run</code> from <code class="highlighter-rouge">IOApp</code> that we’re using. When we now run this application, it’ll start making HTTP requests and stores the result.</p>
<p>We mentioned a couple of times that you can plug in different IO implementations. For instance if you want to run this program using ZIO, the code would look something like this:</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">import</span> <span class="nn">tagless.services.LiveConfigInterpreter</span>
<span class="k">import</span> <span class="nn">tagless.services.http4sServer.LiveHttp4sServerInterpreter</span>
<span class="k">import</span> <span class="nn">scala.concurrent.ExecutionContext</span>
<span class="k">import</span> <span class="nn">tagless.services.repo.storage</span>
<span class="k">import</span> <span class="nn">tagless.services.db.mongo</span>
<span class="k">import</span> <span class="nn">zio._</span>
<span class="k">import</span> <span class="nn">zio.interop.catz.implicits._</span>
<span class="k">import</span> <span class="nn">zio.interop.catz._</span>
<span class="cm">/**
* Simple runner which uses the ZIO monad to run everything
*/</span>
<span class="k">object</span> <span class="nc">RunnerZIO</span> <span class="k">extends</span> <span class="n">zio</span><span class="o">.</span><span class="nc">App</span> <span class="o">{</span>
<span class="k">def</span> <span class="n">run</span><span class="o">(</span><span class="n">args</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">String</span><span class="o">])</span><span class="k">:</span> <span class="kt">URIO</span><span class="o">[</span><span class="kt">ZEnv</span>, <span class="kt">ExitCode</span><span class="o">]</span> <span class="k">=</span> <span class="o">{</span>
<span class="k">type</span> <span class="kt">Task</span><span class="o">[</span><span class="kt">A</span><span class="o">]</span> <span class="k">=</span> <span class="nc">ZIO</span><span class="o">[</span><span class="kt">Any</span>, <span class="kt">Throwable</span>, <span class="kt">A</span><span class="o">]</span>
<span class="c1">// get all the services, and add them to the implicit scope
</span> <span class="k">implicit</span> <span class="k">val</span> <span class="n">ec</span> <span class="k">=</span> <span class="nc">ExecutionContext</span><span class="o">.</span><span class="n">global</span>
<span class="c1">// the concrete implementations mapped to the IO monad
</span> <span class="k">implicit</span> <span class="k">val</span> <span class="n">configService</span> <span class="k">=</span> <span class="nc">LiveConfigInterpreter</span><span class="o">[</span><span class="kt">Task</span><span class="o">]()</span>
<span class="k">implicit</span> <span class="k">val</span> <span class="n">http4sClient</span> <span class="k">=</span> <span class="n">services</span><span class="o">.</span><span class="n">http4sClient</span><span class="o">.</span><span class="nc">LiveHttp4sClientInterpreter</span><span class="o">[</span><span class="kt">Task</span><span class="o">]()</span>
<span class="k">implicit</span> <span class="k">val</span> <span class="nc">MongoDBConnectionDSL</span> <span class="k">=</span> <span class="n">mongo</span><span class="o">.</span><span class="nc">LiveMongoDBConnectionInterpreter</span><span class="o">[</span><span class="kt">Task</span><span class="o">]()</span>
<span class="k">val</span> <span class="n">http4s</span> <span class="k">=</span> <span class="nc">LiveHttp4sServerInterpreter</span><span class="o">[</span><span class="kt">Task</span><span class="o">]()</span>
<span class="k">implicit</span> <span class="k">val</span> <span class="n">tempClient</span> <span class="k">=</span> <span class="n">services</span><span class="o">.</span><span class="n">tempClient</span><span class="o">.</span><span class="nc">LiveTempClientInterpreter</span><span class="o">[</span><span class="kt">Task</span><span class="o">]()</span>
<span class="k">val</span> <span class="n">tempStorage</span> <span class="k">=</span> <span class="n">storage</span><span class="o">.</span><span class="nc">TemperatureStorageInterpreter</span><span class="o">[</span><span class="kt">Task</span><span class="o">]()</span>
<span class="c1">// define the tasks to run in parallel
</span> <span class="k">val</span> <span class="n">streamTask</span> <span class="k">=</span> <span class="n">tempClient</span><span class="o">.</span><span class="n">temperatureStream</span><span class="o">.</span><span class="n">flatMap</span> <span class="o">{</span> <span class="k">_</span><span class="o">.</span><span class="n">evalMap</span><span class="o">(</span><span class="n">tempStorage</span><span class="o">.</span><span class="n">insert</span><span class="o">).</span><span class="n">compile</span><span class="o">.</span><span class="n">drain</span> <span class="o">}</span>
<span class="k">val</span> <span class="n">http4sServerTask</span> <span class="k">=</span> <span class="n">http4s</span><span class="o">.</span><span class="n">server</span><span class="o">.</span><span class="n">toManagedZIO</span><span class="o">.</span><span class="n">useForever</span>
<span class="c1">// run the tasks in parallel
</span> <span class="nc">Task</span><span class="o">.</span><span class="n">collectAllPar</span><span class="o">(</span><span class="nc">Set</span><span class="o">(</span><span class="n">streamTask</span><span class="o">,</span> <span class="n">http4sServerTask</span><span class="o">)).</span><span class="n">exitCode</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Note that there are different ways of running tasks in parallel. I just used <code class="highlighter-rouge">collectAllPar</code> here.</p>
<h2 id="conclusion">conclusion</h2>
<p>Converting the ZIO based approach from the previous articles to a cats-effect approach where we inject dependend modules using implicit constraints was fairly straightforward. The effects from cats-effect provide sufficient functionality for the implementation. This is however a fairly simple application, so I wonder a bit what is to gain from having the <code class="highlighter-rouge">F[_]</code> abstraction instead of choosing an IO implementation beforehand and sticking with that one.
The way dependencies are injected feels rather nice with the constraint approach, it makes it very clear what is needed where and at least gives an idea what the implementation are going to do, instead of just providing a big IO and giving the modules access to everything. I do miss the layer abstraction from ZIO a bit since that felt like quite a nice abstraction on top to manage dependencies.</p>
<p>In the end, however, I think Cats Effect and the Cats IO monad, ZIO, Monix and probably some other stuff I’ve forgotten show how great the FP and effects support is in Scala. When building libraries I really see the big advantage of not restricting yourself to a single IO implementation. For developing business logic, or functionality that doesn’t need to be shared, I wonder whether abstracting away the IO implementation is that useful, since it doesn’t allow direct access to the great stuff from the Cats IO monad or the ZIO implementations.</p>
<p>Nevertheless, been a great experience implementing this, and I keep finding it amazing how far Scala has come from just a couple of years ago!</p>Jos DirksenIn two previous articles we explored how you can use ZIO to easily create programs in a type-safe manner: Exploring ZIO - Part I Exploring ZIO - Part II - ZStream and modules In those articles we created a simple server which polls a REST endpoint at a fixed interval, and wrote the results in a database (MongoDB). The data was then exposed through an HTTP4S REST endpoint. In that approach we used the ZIO monad throughout the stack, which allowed us to use (in those examples only a small number) of specific ZIO features to make our lives easier. If you want to jump directly to the code you can find that here: https://github.com/josdirksen/tagless-playground Another common approach for creating modules and services is by using the “Tagless Final” pattern. If you search around you’ll find numerous explanations about what this pattern does, but the main idea behind this pattern is: Defer choosing a IO implementation. So our services and traits are defined using F[_] as the type. Dependencies for a certain function (or module) are defined as implicit parameters. E.g like F[_] : ConfigDSL, means that this function (or case class) requires a type F[_] for which a ConfigDSL[F] implementation is in the implicit scope. There are a number of other features in this pattern, but the above captures most of this. There are a couple of libraries that adopt this pattern, where dependencies need to be provided through implicits, and many of those use the generic Cats Effect library to define fine grained effects that functions need. In this article we’ll convert the setup from Exploring ZIO - Part II - ZStream and modules to one where we use Cats Effect together with (part of) the “Tagless Final” pattern. Before we start a quick summary of what we’re going to build: Read a configuration file using PureConfig Use this configuration to create a HTTP4S server and HTTP4S client Create a stream using FS2 which is used to get the latest temperatures from a remote server. Store the results from the external client in MongoDB. All the steps above will be defined using ‘Cats Effect’ and we’ll only provide a concrete IO monad implementation at the edge of the world in our main function. Loading configuration The main difference with the ZIO approach is that we define the trait using a F[_] case class Config(apiConfig: ApiConfig, temperatureConfig: TemperatureConfig, dbConfig: DBConfig) case class ApiConfig(endpoint: String, port: Int) case class TemperatureConfig(endpoint: String, apiKey: String, interval: FiniteDuration) case class DBConfig(endpoint: String) /** * Basic DSL trait is type, constraint agnostic. */ trait ConfigDSL[F[_]] { val config: F[Config] } object ConfigDSL { def apply[F[_]](implicit F: ConfigDSL[F]): F.type = F } To use this config we’re going to define an interpreter for it. While we can have multiple interpreters, we should preferably only have one. A simple implementation using PureConfig is shown here. Note that in this implementation we still don’t decide on the actual IO monad we’re going to run the final program in. That’s something we decide at the latest possible time. /** * A pureconfig based interpreter. Note that we use the kind projector, so we can * correctly specify the MonadError constraint. If we don't use this we have to * either add it as a desugared implicit, or create a custom type. */ case class LiveConfigInterpreter[F[_]: MonadError[*[_], Throwable]]() extends ConfigDSL[F] { override val config: F[Config] = { /** * We can wrap it in an either, if we do that then we don't * really need any other typeclasses, we can also say that * we expect the result to be mapped to a monadError. That * means that the F[_] used, should satisfy the MonadError * constraint. */ val c = ConfigSource.default .load[Config] .left .map(pureconfig.error.ConfigReaderException.apply) // summon the monadError instance and convert to an either MonadError[F, Throwable].fromEither(c) } } In the code above we want to communicate to the users that loading this config can fail. So we expect that the monad the user provides supports at least the functionality defined in MonadError. This could for instance be an Either or a Try. We define this dependency as a constraint on F[_] like this: F[_]: MonadError[*[_], Throwable]. This simpler syntax is provided by the kind project (https://github.com/typelevel/kind-projector), which allows for easier syntax. This is enabled in the build.sbt: ThisBuild / scalaVersion := "2.13.4" ThisBuild / version := "0.1.0-SNAPSHOT" ThisBuild / organization := "org.smartjava" ThisBuild / organizationName := "smartjava" val CatsVersion = "3.1.1" val Http4sVersion = "1.0.0-M23" lazy val hello = (project in file(".")) .settings( name := "Exploring Tagless", libraryDependencies ++= Seq( "org.typelevel" %% "cats-effect" % CatsVersion, "com.github.pureconfig" %% "pureconfig" % "0.15.0", "org.http4s" %% "http4s-blaze-server" % Http4sVersion, "org.http4s" %% "http4s-dsl" % Http4sVersion, "org.http4s" %% "http4s-blaze-client" % Http4sVersion, "org.http4s" %% "http4s-circe" % Http4sVersion, "ch.qos.logback" % "logback-core" % "1.2.3", "org.slf4j" % "slf4j-api" % "1.7.30", "ch.qos.logback" % "logback-classic" % "1.2.3", "co.fs2" %% "fs2-core" % "3.0.4", "co.fs2" %% "fs2-io" % "3.0.4", "co.fs2" %% "fs2-reactive-streams" % "3.0.4", // JSON Mapping "io.circe" %% "circe-generic" % "0.12.3", "io.circe" %% "circe-literal" % "0.12.3", "org.mongodb.scala" %% "mongo-scala-driver" % "4.2.3" ), scalacOptions += "-Ymacro-annotations" ) addCompilerPlugin("org.typelevel" % "kind-projector" % "0.11.3" cross CrossVersion.full) In the final main we’ll use this configuration something like this: import cats.effect.IO implicit val configService = LiveConfigInterpreter[IO]() With the configuration out of the way we can look at configuring our HTTP4S client and server Setting up HTTP4s We’ll start by looking at the client. object http4sClient { /** * Simple DSL to get an instance of a client */ trait Http4sClientDSL[F[_]] { val client: Resource[F, Client[F]] } object Http4sClientDSL { def apply[F[_]](implicit F: Http4sClientDSL[F]): F.type = F } /** * Live version */ case class LiveHttp4sClientInterpreter[F[_]: Async]()(implicit ec: ExecutionContext) extends Http4sClientDSL[F] { override val client: Resource[F, Client[F]] = BlazeClientBuilder[F](ec).resource } } If you look at the trait, we’re going to return a Resource[F, Client[F]]. Which basically allows us to get access to a Client[F] by calling use on the resource. Since HTTP4s already is based on Cats Effect, we only have to call BlazeClientBuilder[F](ec).resource and we’re done… well almost. If you look at the signature: def apply[F[_]: Async](executionContext: ExecutionContext): BlazeClientBuilder[F] = This function requires an Async[F] to be in scope. So that’s why we also added it to the case class (which serves as our module / service). You can also see that this function requires an explicit execution context. It seems to be rather arbitrary when the ExectionContext is passed explicitly or implicitly, so in our module we expect this to be passed in implicitly. The Http4sServer setup isn’t that different: object http4sServer { /** * This trait provides access to a Http4sServer dependency. It is * wrapped in the F monad, so we can signal errors during setup. * * Wondering whether these should be called DSL, since these are * just the low level dependencies. */ trait Http4sServerDSL[F[_]] { val server: Resource[F, Server] } /** * Helper object to access the instance of the DSL that is in scope */ object Http4ServerDSL { def apply[F[_]](implicit F: Http4sServerDSL[F]): F.type = F } case class routes[F[_]: Monad]() { private val dsl = Http4sDsl[F] import dsl._ val routes: HttpRoutes[F] = HttpRoutes .of[F] { case GET -> Root / "users" / IntVar(id) => { Created("A value here") } case POST -> Root / "users" => { Created("And another one here") } } } /** * Live instance of the HttpsServer service. * * @param ec */ case class LiveHttp4sServerInterpreter[ F[_]: MonadCancel[*[_], Throwable]: Async: ConfigDSL ]()(implicit ec: ExecutionContext) extends Http4sServerDSL[F] { override val server: Resource[F, Server] = for { config <- Resource.eval(ConfigDSL[F].config) server <- BlazeServerBuilder[F](ec) .bindHttp(config.apiConfig.port, config.apiConfig.endpoint) .withHttpApp(routes[F]().routes.orNotFound) .resource } yield ( server ) } } As you can see in the code above, the process is the same. We pass in a couple of effects from Cats Effect (MonadCancel and Async) since those are required by BlazeServerBuilder[F]. Additionally we also add ConfigDSL to the list of constraints. This means that to create this module, we need to make sure a ConfigDSL[F] is in scope as well. If we want to instantiate this module we’d need to do something like this: implicit val ec = ExecutionContext.global implicit val configService = LiveConfigInterpreter[IO]() // implicit since it'll be used a dependency later on. implicit val http4sClient = services.http4sClient.LiveHttp4sClientInterpreter[IO]() val http4s = LiveHttp4sServerInterpreter[IO]() That way we can easily get access to the Http4S server. Configuring MongoDB for data storage Now let’s look at the bottom layer of the application where we want to store data in a mongo database. First we’ll look at how to acquire the mongo client: object mongo { trait MongoDBConnectionDSL[F[_]] { val mongoClient: Resource[F, MongoClient] } object MongoDBConnectionDSL { def apply[F[_]](implicit F: MongoDBConnectionDSL[F]): F.type = F } case class LiveMongoDBConnectionInterpreter[F[_]: Sync: ConfigDSL]() extends MongoDBConnectionDSL[F] { val ME = MonadError[F, Throwable] val Config = ConfigDSL[F] override val mongoClient = Resource.make(acquireConnection)(releaseConnection) /** * Acquire a connection */ private def acquireConnection()(implicit config: ConfigDSL[F]) = for { c <- config.config r <- Sync[F].blocking(MongoClient(c.dbConfig.endpoint)) } yield { r } /** * Release the mongo client, wrap in sync since it's effectful */ private def releaseConnection(mongoClient: MongoClient) = Sync[F].blocking(mongoClient.close) } } Here you can again see a very simple trait explaining what this DSL offers (not much in this simple example), and an implementation (still agnostic on F[_]). The only implementation here is that we require a Sync[F] implementation, which we use to call the effectfull instantiation of the MongoClient. Since we do this as a resource we have to provide a function to acquire the resource acquireConnection and a function to release the resource again releaseConnection. Using Sync[F] will defer the code within the blocking function until the effect is evaluated. Now that we’ve got a way to access the mongo client, we can also implement storing and retrieving our domain entity (a Temperature object). The code below is almost exactly the same as it was in the ZIO example with ZStream, so we won’t explain the details: object storage { trait TemperatureStorageDSL[F[_]] { def insert(temperature: Temperature): F[Unit] def getAll(): F[List[Temperature]] } object TemperatureStorageDSL { def apply[F[_]](implicit F: TemperatureStorageDSL[F]): F.type = F } case class TemperatureStorageInterpreter[ F[_]: MonadCancel[*[_], Throwable]: Async: ConfigDSL: MongoDBConnectionDSL ]() extends TemperatureStorageDSL[F] { val temperatureCodecProvider = Macros.createCodecProvider[Temperature]() val codecRegistry = fromRegistries(fromProviders(temperatureCodecProvider), DEFAULT_CODEC_REGISTRY) val ME = MonadError[F, Throwable] override def insert(temperature: Temperature): F[Unit] = withCollection[Unit, Temperature] { c => c.insertOne(temperature) .toStream[F] .compile .toList .map(_.length) // naively assume that when we get 1 insert result everything is fine .flatMap { case 1 => ME.pure() case _ => ME.raiseError[Unit](new IllegalArgumentException("Expected result from mongodb")) } } override def getAll(): F[List[Temperature]] = { withCollection[List[Temperature], Temperature] { _.find() .toStream[F] .compile .toList } } /** * Get the database and collection to which to store. * * @param f function to call within the context of this collection * @return result of wrapped */ private def withCollection[A, T: ClassTag]( f: MongoCollection[T] => F[A] ): F[A] = MongoDBConnectionDSL[F].mongoClient.use[A] { mongoClient => val collection = mongoClient .getDatabase("sampleservice") .withCodecRegistry(codecRegistry) .getCollection[T]("temperatures") f(collection) } } } The only interesting part here are the dependencies we want for this module: case class TemperatureStorageInterpreter[ F[_]: MonadCancel[*[_], Throwable]: Async: ConfigDSL: MongoDBConnectionDSL ]() extends TemperatureStorageDSL[F] { So what we need are a couple of classes from Cats Effect, we require an instance of the ConfigDSL and we require an instance of the MongoDBConnectionDSL as well. And of course since the client is a resource we have to use use to access the client in a safe manner: private def withCollection[A, T: ClassTag]( f: MongoCollection[T] => F[A] ): F[A] = MongoDBConnectionDSL[F].mongoClient.use[A] { mongoClient => val collection = mongoClient .getDatabase("sampleservice") .withCodecRegistry(codecRegistry) .getCollection[T]("temperatures") f(collection) } When this effect is evaluated, the acquire function we defined for retrieving a MongoClient will be called, and at the end of the function the release function we defined is called. And the final part that is interesting is the integration we have here with FS2. The mongo client we use is based on the Reactive Streams implementation. Luckily we can just integrate with that directly with FS2: c.insertOne(temperature) .toStream[F] That will result in a monad agnostic stream (def toStream[F[_]: Async]: Stream[F, A]) and the only thing we need to do is make sure the monad we provide has the Async constraint. Create a stream of temperature updates The final thing we need to do is create a client that polls an http endpoint, and returns a stream of temperature updates. For that we just define a new module where we expose a single value which is a stream of temperatures. The type is defined like this: F[Stream[F, Temperature]]. This means that retrieving the stream itself is also effectful (since we use information from the configuration) and when we evaluate the stream we also do this within the context of the supplied F[_]. So our stream itself uses functions which aren’t pure (e.g simply calculations), in this case we make a call using the passed in Http4sClientDSL: object tempClient { trait TempClientDSL[F[_]] { val temperatureStream: F[Stream[F, Temperature]] } case class LiveTempClientInterpreter[F[_]: MonadError[*[_], Throwable]: Temporal: ConfigDSL: Http4sClientDSL]() extends TempClientDSL[F] { import org.http4s.circe._ import io.circe.generic.auto._ import TempClientLive.OpenWeather._ implicit val userDecoder = jsonOf[F, OWResult] /** * Get a stream in an effectful way. */ override val temperatureStream: F[Stream[F, Temperature]] = for { config <- ConfigDSL[F].config stream = (Stream(1) ++ Stream.awakeEvery(config.temperatureConfig.interval)).evalMap { _ => makeTemperatureCall(config.temperatureConfig.endpoint + config.temperatureConfig.apiKey) } } yield { stream } /** * Make an actual call to a rest endpoint */ private def makeTemperatureCall(url: String)(implicit clientDSL: Http4sClientDSL[F]): F[Temperature] = clientDSL.client.use { _.expect[TempClientLive.OpenWeather.OWResult](url).map(t => Temperature(t.dt, t.main.temp)) } } object TempClientLive { object OpenWeather { // the openweather model case class OWResult(coord: OWCoord, main: OWMain, visibility: Integer, wind: OWWind, dt: Long) case class OWCoord(lat: Double, lon: Double) case class OWMain( temp: Double, feels_like: Double, temp_min: Double, temp_max: Double, pressure: Int, humidity: Int ) case class OWWind(speed: Double, deg: Long, gust: Double) } } } When looking at this implementation you can see that in the case class definition we define a number of dependencies, just like we did for the other modules. The interesting part here is where we actually make and use the FS2 stream: /** * Get a stream in an effectful way. */ override val temperatureStream: F[Stream[F, Temperature]] = for { config <- ConfigDSL[F].config stream = (Stream(1) ++ Stream.awakeEvery(config.temperatureConfig.interval)).evalMap { _ => makeTemperatureCall(config.temperatureConfig.endpoint + config.temperatureConfig.apiKey) } } yield { stream } /** * Make an actual call to a rest endpoint */ private def makeTemperatureCall(url: String)(implicit clientDSL: Http4sClientDSL[F]): F[Temperature] = clientDSL.client.use { _.expect[TempClientLive.OpenWeather.OWResult](url).map(t => Temperature(t.dt, t.main.temp)) } We concatenate two stream together, this means that as soon as we evaluate this effect our stream immediately emits a value, and then it’ll emit values at the specified interval. Whenever a value is emitted, we run an effectfull function using evalMap which uses the Http4sClientDSL[F] to make the actual call. The interesting thing is, that all this is still done without knowing anything (well almost anything) about the F[_] monad. Glue everything together All that is left to do is tie everything together and select the IO monad that we want to use: object Runner extends IOApp { def run(args: List[String]): IO[ExitCode] = { // get all the services, and add them to the implicit scope implicit val ec = ExecutionContext.global // the concrete implementations mapped to the IO monad implicit val configService = LiveConfigInterpreter[IO]() implicit val http4sClient = services.http4sClient.LiveHttp4sClientInterpreter[IO]() implicit val MongoDBConnectionDSL = mongo.LiveMongoDBConnectionInterpreter[IO]() val http4s = LiveHttp4sServerInterpreter[IO]() implicit val tempClient = services.tempClient.LiveTempClientInterpreter[IO]() val tempStorage = storage.TemperatureStorageInterpreter[IO]() val program = for { // consume the stream stream <- tempClient.temperatureStream _ <- stream .evalMap(tempStorage.insert) .compile .drain .start // start the server and keep running server <- http4s.server.use(_ => IO.never).as(ExitCode.Success) } yield { server } // return the program to execute program } } The program above uses the Cats IO monad as the monad to run everything in. What you see here is that the dependencies that are injected are defined as implicit val so they are automatically applied when we create the other modules. The program itself is just a simple for-comprehension which: Gets the stream of temperatures, and drains them. Whenever we receive a new temperature we store it in the database. We use .start to start the stream, since that allows us to start this effect in the background: * Start execution of the source suspended in the `IO` context. * * This can be used for non-deterministic / concurrent execution. Then we start the webserver as well, which we’ll just keep running forever. The result is a val program: IO[ExitCode], which is also the expected type of the def run from IOApp that we’re using. When we now run this application, it’ll start making HTTP requests and stores the result. We mentioned a couple of times that you can plug in different IO implementations. For instance if you want to run this program using ZIO, the code would look something like this: import tagless.services.LiveConfigInterpreter import tagless.services.http4sServer.LiveHttp4sServerInterpreter import scala.concurrent.ExecutionContext import tagless.services.repo.storage import tagless.services.db.mongo import zio._ import zio.interop.catz.implicits._ import zio.interop.catz._ /** * Simple runner which uses the ZIO monad to run everything */ object RunnerZIO extends zio.App { def run(args: List[String]): URIO[ZEnv, ExitCode] = { type Task[A] = ZIO[Any, Throwable, A] // get all the services, and add them to the implicit scope implicit val ec = ExecutionContext.global // the concrete implementations mapped to the IO monad implicit val configService = LiveConfigInterpreter[Task]() implicit val http4sClient = services.http4sClient.LiveHttp4sClientInterpreter[Task]() implicit val MongoDBConnectionDSL = mongo.LiveMongoDBConnectionInterpreter[Task]() val http4s = LiveHttp4sServerInterpreter[Task]() implicit val tempClient = services.tempClient.LiveTempClientInterpreter[Task]() val tempStorage = storage.TemperatureStorageInterpreter[Task]() // define the tasks to run in parallel val streamTask = tempClient.temperatureStream.flatMap { _.evalMap(tempStorage.insert).compile.drain } val http4sServerTask = http4s.server.toManagedZIO.useForever // run the tasks in parallel Task.collectAllPar(Set(streamTask, http4sServerTask)).exitCode } } Note that there are different ways of running tasks in parallel. I just used collectAllPar here. conclusion Converting the ZIO based approach from the previous articles to a cats-effect approach where we inject dependend modules using implicit constraints was fairly straightforward. The effects from cats-effect provide sufficient functionality for the implementation. This is however a fairly simple application, so I wonder a bit what is to gain from having the F[_] abstraction instead of choosing an IO implementation beforehand and sticking with that one. The way dependencies are injected feels rather nice with the constraint approach, it makes it very clear what is needed where and at least gives an idea what the implementation are going to do, instead of just providing a big IO and giving the modules access to everything. I do miss the layer abstraction from ZIO a bit since that felt like quite a nice abstraction on top to manage dependencies. In the end, however, I think Cats Effect and the Cats IO monad, ZIO, Monix and probably some other stuff I’ve forgotten show how great the FP and effects support is in Scala. When building libraries I really see the big advantage of not restricting yourself to a single IO implementation. For developing business logic, or functionality that doesn’t need to be shared, I wonder whether abstracting away the IO implementation is that useful, since it doesn’t allow direct access to the great stuff from the Cats IO monad or the ZIO implementations. Nevertheless, been a great experience implementing this, and I keep finding it amazing how far Scala has come from just a couple of years ago!Exploring ZIO - Part II - ZStream and modules2021-05-24T00:00:00+02:002021-05-24T00:00:00+02:00http://www.smartjava.org/content/exploring-zio-part-2<p>In the <a href="https://www.smartjava.org/content/exploring-zio-part-1/">previous article</a> we looked at the basics of ZIO. In this part we’re going to extend on that example and add the following:</p>
<ul>
<li><a href="https://zio.dev/docs/datatypes/stream/stream">ZStream</a> integration: we’re going to create our own stream that polls an external service, and we’re going to use the reactive-stream integration to query a Mongo database.</li>
<li>Storage layer: we’ll add persistence to the example, where everything we get from the external service is stored in a database (MongoDB in our case).</li>
<li><a href="https://zio.dev/docs/datatypes/contextual/#module-pattern-20">Module pattern 2.0</a>: we’ve rewritten a couple of services to the new module pattern. Not all services could be rewritten though, but I’ll try to explain my reasoning why and why not.</li>
<li>Real REST endpoint: We’ll connect the HTTP4S endpoint to the MongoDB server to actually retrieve some data.</li>
</ul>
<p>We’ll not look at all the code (which you can find here: https://github.com/josdirksen/zio-playground), but focus on the main data flow and how that is connected. The main steps we’ll explore in this article are:</p>
<ol>
<li>Poll an external REST endpoint to retrieve temperature data.</li>
<li>This data is then stored in MongoDB.</li>
<li>Expose a REST endpoint to query all the stored data.</li>
</ol>
<h2 id="getting-started">Getting started</h2>
<p>First of, let’s quickly look at the <code class="highlighter-rouge">sbt</code> dependencies so you get a sense of which libraries we’ll be using:</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">import</span> <span class="nn">Dependencies._</span>
<span class="nc">ThisBuild</span> <span class="o">/</span> <span class="n">scalaVersion</span> <span class="o">:=</span> <span class="s">"2.13.4"</span>
<span class="nc">ThisBuild</span> <span class="o">/</span> <span class="n">version</span> <span class="o">:=</span> <span class="s">"0.1.0-SNAPSHOT"</span>
<span class="nc">ThisBuild</span> <span class="o">/</span> <span class="n">organization</span> <span class="o">:=</span> <span class="s">"org.smartjava"</span>
<span class="nc">ThisBuild</span> <span class="o">/</span> <span class="n">organizationName</span> <span class="o">:=</span> <span class="s">"smartjava"</span>
<span class="c1">// need to use an older version, since the newest version of http4s
// supports the latest cats version, while the zio-interop-cats one
// doesn't support this yet.
</span><span class="k">val</span> <span class="nc">Http4sVersion</span> <span class="k">=</span> <span class="s">"1.0.0-M4"</span>
<span class="k">val</span> <span class="nc">ZioVersion</span> <span class="k">=</span> <span class="s">"1.0.8"</span>
<span class="k">lazy</span> <span class="k">val</span> <span class="n">root</span> <span class="k">=</span> <span class="o">(</span><span class="n">project</span> <span class="n">in</span> <span class="n">file</span><span class="o">(</span><span class="s">"."</span><span class="o">))</span>
<span class="o">.</span><span class="n">settings</span><span class="o">(</span>
<span class="n">name</span> <span class="o">:=</span> <span class="s">"zio-playground"</span><span class="o">,</span>
<span class="n">libraryDependencies</span> <span class="o">+=</span> <span class="n">scalaTest</span> <span class="o">%</span> <span class="nc">Test</span><span class="o">,</span>
<span class="n">libraryDependencies</span> <span class="o">++=</span> <span class="nc">Seq</span><span class="o">(</span>
<span class="s">"org.http4s"</span> <span class="o">%%</span> <span class="s">"http4s-blaze-server"</span> <span class="o">%</span> <span class="nc">Http4sVersion</span><span class="o">,</span>
<span class="s">"org.http4s"</span> <span class="o">%%</span> <span class="s">"http4s-dsl"</span> <span class="o">%</span> <span class="nc">Http4sVersion</span><span class="o">,</span>
<span class="s">"org.http4s"</span> <span class="o">%%</span> <span class="s">"http4s-blaze-client"</span> <span class="o">%</span> <span class="nc">Http4sVersion</span><span class="o">,</span>
<span class="s">"org.http4s"</span> <span class="o">%%</span> <span class="s">"http4s-circe"</span> <span class="o">%</span> <span class="nc">Http4sVersion</span><span class="o">,</span>
<span class="c1">// JSON Mapping
</span> <span class="s">"io.circe"</span> <span class="o">%%</span> <span class="s">"circe-generic"</span> <span class="o">%</span> <span class="s">"0.12.3"</span><span class="o">,</span>
<span class="s">"io.circe"</span> <span class="o">%%</span> <span class="s">"circe-literal"</span> <span class="o">%</span> <span class="s">"0.12.3"</span><span class="o">,</span>
<span class="c1">// ZIO stuff
</span> <span class="s">"dev.zio"</span> <span class="o">%%</span> <span class="s">"zio"</span> <span class="o">%</span> <span class="nc">ZioVersion</span><span class="o">,</span>
<span class="s">"dev.zio"</span> <span class="o">%%</span> <span class="s">"zio-streams"</span> <span class="o">%</span> <span class="nc">ZioVersion</span><span class="o">,</span>
<span class="s">"dev.zio"</span> <span class="o">%%</span> <span class="s">"zio-interop-reactivestreams"</span> <span class="o">%</span> <span class="s">"1.3.5"</span><span class="o">,</span>
<span class="s">"dev.zio"</span> <span class="o">%%</span> <span class="s">"zio-interop-cats"</span> <span class="o">%</span> <span class="s">"2.4.0.0"</span><span class="o">,</span>
<span class="s">"com.github.pureconfig"</span> <span class="o">%%</span> <span class="s">"pureconfig"</span> <span class="o">%</span> <span class="s">"0.15.0"</span><span class="o">,</span>
<span class="s">"org.slf4j"</span> <span class="o">%</span> <span class="s">"slf4j-api"</span> <span class="o">%</span> <span class="s">"1.7.5"</span><span class="o">,</span>
<span class="s">"org.slf4j"</span> <span class="o">%</span> <span class="s">"slf4j-simple"</span> <span class="o">%</span> <span class="s">"1.7.5"</span><span class="o">,</span>
<span class="s">"dev.zio"</span> <span class="o">%%</span> <span class="s">"zio-test"</span> <span class="o">%</span> <span class="nc">ZioVersion</span> <span class="o">%</span> <span class="s">"test"</span><span class="o">,</span>
<span class="s">"dev.zio"</span> <span class="o">%%</span> <span class="s">"zio-test-sbt"</span> <span class="o">%</span> <span class="nc">ZioVersion</span> <span class="o">%</span> <span class="s">"test"</span><span class="o">,</span>
<span class="s">"org.mongodb.scala"</span> <span class="o">%%</span> <span class="s">"mongo-scala-driver"</span> <span class="o">%</span> <span class="s">"4.2.3"</span>
<span class="o">),</span>
<span class="n">testFrameworks</span> <span class="o">+=</span> <span class="k">new</span> <span class="nc">TestFramework</span><span class="o">(</span><span class="s">"zio.test.sbt.ZTestFramework"</span><span class="o">)</span>
<span class="o">)</span>
</code></pre></div></div>
<p>Nothing to special. We use <code class="highlighter-rouge">circe</code> for the JSON stuff, use the standard <code class="highlighter-rouge">mongo-scala-driver</code> for accessing the database, and we’ve addedd the relevant <code class="highlighter-rouge">zio-streams</code> dependencies to create streams and to interact with <code class="highlighter-rouge">reactivestreams</code>.</p>
<h2 id="creating-our-first-stream">Creating our first stream</h2>
<p>We’re first going to create our own stream. This stream will be a stream of <code class="highlighter-rouge">Temperature</code> values retrieved from an external API. This is a stream that’ll poll the external service every so often, and pass on these values to anyone that is interested.</p>
<p>All the stuff happens in the <code class="highlighter-rouge">TempClient.scala</code> file (we’ll explain the interesting parts below this code fragment):</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cm">/** The temperature client calls a remote webservice ever so often and gives
* access to a stream of temperatures.
*/</span>
<span class="k">object</span> <span class="nc">tempClient</span> <span class="o">{</span>
<span class="k">trait</span> <span class="nc">TempClient</span> <span class="o">{</span>
<span class="k">val</span> <span class="n">temperatureStream</span><span class="k">:</span> <span class="kt">ZStream</span><span class="o">[</span><span class="kt">Any</span>, <span class="kt">Throwable</span>, <span class="kt">Temperature</span><span class="o">]</span>
<span class="o">}</span>
<span class="k">object</span> <span class="nc">TempClient</span> <span class="o">{</span>
<span class="cm">/** get the zstream within the context of the provided environment. This will
* return a stream that, when drained, will provide a temperature update every
* tick. The only dependency here is the TempClient, retrieving this stream
* won't result in any errors. Note that we lift the temperatureStream in
* the UIO, else we can't use the serviceWith function, which expects the
* result to be A wrapped in a URIO
*/</span>
<span class="k">val</span> <span class="n">temperatureStream</span><span class="k">:</span> <span class="kt">URIO</span><span class="o">[</span><span class="kt">Has</span><span class="o">[</span><span class="kt">TempClient</span><span class="o">]</span>, <span class="kt">ZStream</span><span class="o">[</span><span class="kt">Any</span>, <span class="kt">Throwable</span>, <span class="kt">Temperature</span><span class="o">]]</span> <span class="k">=</span>
<span class="nc">ZIO</span><span class="o">.</span><span class="n">serviceWith</span><span class="o">[</span><span class="kt">TempClient</span><span class="o">](</span><span class="n">s</span> <span class="k">=></span> <span class="nc">UIO</span><span class="o">.</span><span class="n">succeed</span><span class="o">(</span><span class="n">s</span><span class="o">.</span><span class="n">temperatureStream</span><span class="o">))</span>
<span class="o">}</span>
<span class="k">case</span> <span class="k">class</span> <span class="nc">TempClientLive</span><span class="o">(</span>
<span class="n">client</span><span class="k">:</span> <span class="kt">Client</span><span class="o">[</span><span class="kt">Task</span><span class="o">],</span>
<span class="n">console</span><span class="k">:</span> <span class="kt">Console.Service</span><span class="o">,</span>
<span class="n">clock</span><span class="k">:</span> <span class="kt">Clock.Service</span><span class="o">,</span>
<span class="n">configuration</span><span class="k">:</span> <span class="kt">Configuration</span>
<span class="o">)</span> <span class="k">extends</span> <span class="nc">TempClient</span> <span class="o">{</span>
<span class="k">import</span> <span class="nn">org.http4s.circe._</span>
<span class="k">import</span> <span class="nn">io.circe.generic.auto._</span>
<span class="k">import</span> <span class="nn">TempClientLive.OpenWeather._</span>
<span class="k">implicit</span> <span class="k">val</span> <span class="n">userDecoder</span> <span class="k">=</span> <span class="n">jsonOf</span><span class="o">[</span><span class="kt">Task</span>, <span class="kt">OWResult</span><span class="o">]</span>
<span class="k">val</span> <span class="n">tempConfig</span> <span class="k">=</span> <span class="n">configuration</span><span class="o">.</span><span class="n">config</span><span class="o">.</span><span class="n">temperatureConfig</span>
<span class="k">override</span> <span class="k">val</span> <span class="n">temperatureStream</span><span class="k">:</span> <span class="kt">ZStream</span><span class="o">[</span><span class="kt">Any</span>, <span class="kt">Throwable</span>, <span class="kt">Temperature</span><span class="o">]</span> <span class="k">=</span>
<span class="c1">// emit one right away and then do the rest in an interval
</span> <span class="o">(</span><span class="nc">ZStream</span><span class="o">.</span><span class="n">succeed</span><span class="o">(-</span><span class="mi">1L</span><span class="o">)</span> <span class="o">++</span> <span class="nc">ZStream</span><span class="o">.</span><span class="n">fromSchedule</span><span class="o">(</span><span class="nc">Schedule</span><span class="o">.</span><span class="n">spaced</span><span class="o">(</span><span class="n">tempConfig</span><span class="o">.</span><span class="n">interval</span><span class="o">.</span><span class="n">toJava</span><span class="o">)))</span>
<span class="o">.</span><span class="n">mapM</span><span class="o">(</span><span class="k">_</span> <span class="k">=></span> <span class="n">makeTemperatureCall</span><span class="o">(</span><span class="n">tempConfig</span><span class="o">.</span><span class="n">endpoint</span> <span class="o">+</span> <span class="n">tempConfig</span><span class="o">.</span><span class="n">apiKey</span><span class="o">))</span>
<span class="o">.</span><span class="n">tap</span> <span class="o">{</span> <span class="n">p</span> <span class="k">=></span> <span class="n">console</span><span class="o">.</span><span class="n">putStrLn</span><span class="o">(</span><span class="n">p</span><span class="o">.</span><span class="n">toString</span><span class="o">())</span> <span class="o">}</span>
<span class="c1">// We still have to provide the clock for the schedule to eliminate all R
</span> <span class="c1">// and just use everything provided to this service.
</span> <span class="o">.</span><span class="n">provide</span><span class="o">(</span><span class="nc">Has</span><span class="o">(</span><span class="n">clock</span><span class="o">))</span>
<span class="cm">/** Make the call. This requires a client to be in the environment, and returns a string
*/</span>
<span class="k">private</span> <span class="k">def</span> <span class="n">makeTemperatureCall</span><span class="o">(</span><span class="n">url</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span><span class="k">:</span> <span class="kt">ZIO</span><span class="o">[</span><span class="kt">Any</span>, <span class="kt">Throwable</span>, <span class="kt">Temperature</span><span class="o">]</span> <span class="k">=</span> <span class="o">{</span>
<span class="k">for</span> <span class="o">{</span>
<span class="c1">// for converting to string, we can use the standard EntityDecoder from HTTP4S together with
</span> <span class="c1">// the zio.interop.cats_ for mapping the Cats stuff to ZIO
</span> <span class="n">res</span> <span class="k"><-</span> <span class="n">client</span><span class="o">.</span><span class="n">expect</span><span class="o">[</span><span class="kt">OWResult</span><span class="o">](</span><span class="n">url</span><span class="o">)</span>
<span class="o">}</span> <span class="k">yield</span> <span class="o">(</span><span class="nc">Temperature</span><span class="o">(</span><span class="n">res</span><span class="o">.</span><span class="n">dt</span><span class="o">,</span> <span class="n">res</span><span class="o">.</span><span class="n">main</span><span class="o">.</span><span class="n">temp</span><span class="o">))</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="k">object</span> <span class="nc">TempClientLive</span> <span class="o">{</span>
<span class="k">object</span> <span class="nc">OpenWeather</span> <span class="o">{</span>
<span class="c1">// the openweather model
</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">OWResult</span><span class="o">(</span><span class="n">coord</span><span class="k">:</span> <span class="kt">OWCoord</span><span class="o">,</span> <span class="n">main</span><span class="k">:</span> <span class="kt">OWMain</span><span class="o">,</span> <span class="n">visibility</span><span class="k">:</span> <span class="kt">Integer</span><span class="o">,</span> <span class="n">wind</span><span class="k">:</span> <span class="kt">OWWind</span><span class="o">,</span> <span class="n">dt</span><span class="k">:</span> <span class="kt">Long</span><span class="o">)</span>
<span class="k">case</span> <span class="k">class</span> <span class="nc">OWCoord</span><span class="o">(</span><span class="n">lat</span><span class="k">:</span> <span class="kt">Double</span><span class="o">,</span> <span class="n">lon</span><span class="k">:</span> <span class="kt">Double</span><span class="o">)</span>
<span class="k">case</span> <span class="k">class</span> <span class="nc">OWMain</span><span class="o">(</span>
<span class="n">temp</span><span class="k">:</span> <span class="kt">Double</span><span class="o">,</span>
<span class="n">feels_like</span><span class="k">:</span> <span class="kt">Double</span><span class="o">,</span>
<span class="n">temp_min</span><span class="k">:</span> <span class="kt">Double</span><span class="o">,</span>
<span class="n">temp_max</span><span class="k">:</span> <span class="kt">Double</span><span class="o">,</span>
<span class="n">pressure</span><span class="k">:</span> <span class="kt">Int</span><span class="o">,</span>
<span class="n">humidity</span><span class="k">:</span> <span class="kt">Int</span>
<span class="o">)</span>
<span class="k">case</span> <span class="k">class</span> <span class="nc">OWWind</span><span class="o">(</span><span class="n">speed</span><span class="k">:</span> <span class="kt">Double</span><span class="o">,</span> <span class="n">deg</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span> <span class="n">gust</span><span class="k">:</span> <span class="kt">Double</span><span class="o">)</span>
<span class="o">}</span>
<span class="c1">// the dependencies for this service
</span> <span class="k">type</span> <span class="kt">TempClientLiveDeps</span> <span class="o">=</span> <span class="nc">Has</span><span class="o">[</span><span class="kt">Client</span><span class="o">[</span><span class="kt">Task</span><span class="o">]]</span>
<span class="k">with</span> <span class="nc">Has</span><span class="o">[</span><span class="kt">Console.Service</span><span class="o">]</span>
<span class="k">with</span> <span class="nc">Has</span><span class="o">[</span><span class="kt">Clock.Service</span><span class="o">]</span>
<span class="k">with</span> <span class="nc">Has</span><span class="o">[</span><span class="kt">Configuration</span><span class="o">]</span>
<span class="c1">// the layer that can be fed to other services, and which specifies what is needed by this layer
</span> <span class="k">val</span> <span class="n">layer</span><span class="k">:</span> <span class="kt">URLayer</span><span class="o">[</span><span class="kt">TempClientLiveDeps</span>, <span class="kt">Has</span><span class="o">[</span><span class="kt">TempClient</span><span class="o">]]</span> <span class="k">=</span> <span class="o">(</span><span class="nc">TempClientLive</span><span class="o">(</span><span class="k">_</span><span class="o">,</span> <span class="k">_</span><span class="o">,</span> <span class="k">_</span><span class="o">,</span> <span class="k">_</span><span class="o">)).</span><span class="n">toLayer</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Let’s start by looking at the structure. This setup uses the <a href="https://zio.dev/docs/datatypes/contextual/#module-pattern-20">Module pattern 2.0</a>:</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">object</span> <span class="nc">tempClient</span> <span class="o">{</span>
<span class="k">trait</span> <span class="nc">TempClient</span> <span class="o">{</span>
<span class="k">val</span> <span class="n">temperatureStream</span><span class="k">:</span> <span class="kt">ZStream</span><span class="o">[</span><span class="kt">Any</span>, <span class="kt">Throwable</span>, <span class="kt">Temperature</span><span class="o">]</span>
<span class="o">}</span>
<span class="k">object</span> <span class="nc">TempClient</span> <span class="o">{</span>
<span class="k">val</span> <span class="n">temperatureStream</span><span class="k">:</span> <span class="kt">URIO</span><span class="o">[</span><span class="kt">Has</span><span class="o">[</span><span class="kt">TempClient</span><span class="o">]</span>, <span class="kt">ZStream</span><span class="o">[</span><span class="kt">Any</span>, <span class="kt">Throwable</span>, <span class="kt">Temperature</span><span class="o">]]</span> <span class="k">=</span>
<span class="nc">ZIO</span><span class="o">.</span><span class="n">serviceWith</span><span class="o">[</span><span class="kt">TempClient</span><span class="o">](</span><span class="n">s</span> <span class="k">=></span> <span class="nc">UIO</span><span class="o">.</span><span class="n">succeed</span><span class="o">(</span><span class="n">s</span><span class="o">.</span><span class="n">temperatureStream</span><span class="o">))</span>
<span class="o">}</span>
<span class="k">case</span> <span class="k">class</span> <span class="nc">TempClientLive</span><span class="o">(</span>
<span class="n">client</span><span class="k">:</span> <span class="kt">Client</span><span class="o">[</span><span class="kt">Task</span><span class="o">],</span>
<span class="n">console</span><span class="k">:</span> <span class="kt">Console.Service</span><span class="o">,</span>
<span class="n">clock</span><span class="k">:</span> <span class="kt">Clock.Service</span><span class="o">,</span>
<span class="n">configuration</span><span class="k">:</span> <span class="kt">Configuration</span>
<span class="o">)</span> <span class="k">extends</span> <span class="nc">TempClient</span> <span class="o">{</span>
<span class="o">...</span>
<span class="o">}</span>
<span class="k">object</span> <span class="nc">TempClientLive</span> <span class="o">{</span>
<span class="o">...</span>
<span class="c1">// the dependencies for this service
</span> <span class="k">type</span> <span class="kt">TempClientLiveDeps</span> <span class="o">=</span> <span class="nc">Has</span><span class="o">[</span><span class="kt">Client</span><span class="o">[</span><span class="kt">Task</span><span class="o">]]</span>
<span class="k">with</span> <span class="nc">Has</span><span class="o">[</span><span class="kt">Console.Service</span><span class="o">]</span>
<span class="k">with</span> <span class="nc">Has</span><span class="o">[</span><span class="kt">Clock.Service</span><span class="o">]</span>
<span class="k">with</span> <span class="nc">Has</span><span class="o">[</span><span class="kt">Configuration</span><span class="o">]</span>
<span class="c1">// the layer that can be fed to other services, and which specifies what is needed by this layer
</span> <span class="k">val</span> <span class="n">layer</span><span class="k">:</span> <span class="kt">URLayer</span><span class="o">[</span><span class="kt">TempClientLiveDeps</span>, <span class="kt">Has</span><span class="o">[</span><span class="kt">TempClient</span><span class="o">]]</span> <span class="k">=</span> <span class="o">(</span><span class="nc">TempClientLive</span><span class="o">(</span><span class="k">_</span><span class="o">,</span> <span class="k">_</span><span class="o">,</span> <span class="k">_</span><span class="o">,</span> <span class="k">_</span><span class="o">)).</span><span class="n">toLayer</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Follow the previous link for a detailled explanation of this pattern, but the main thing that changed is that we don’t introduce a type alias for the <code class="highlighter-rouge">Has[A]</code> part, but just define our module as a simple trait <code class="highlighter-rouge">trait TempClient</code>, and implementations of this trait <code class="highlighter-rouge">TempClientLive</code> are exposed as layers. In our example you can see that our implementation has dependencies to four other modules (<code class="highlighter-rouge">Has[Client[Task]</code>, <code class="highlighter-rouge">Has[Console.Service]</code>, <code class="highlighter-rouge">Has[Clock.Service]</code>, and <code class="highlighter-rouge">Has[Configuration]</code>). You’ll also notice the change here, that we don’t refer types anymore, but explicitly reference the <code class="highlighter-rouge">Has[A]</code> type. I like this, since it makes it much more clear that we’re talking about layers and dependencies, and not actual implementations or traits. Since our implementation now is a simple case class we can use the <code class="highlighter-rouge">toLayer</code> function to automatically inject the dependencies our service needs when creating the layer.</p>
<p>We’ve already looked at the layers in the previous article, so the only interesting part that is left here is the creation of the stream in the <code class="highlighter-rouge">TempClientLive</code> implementation:</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">case</span> <span class="k">class</span> <span class="nc">TempClientLive</span><span class="o">(</span>
<span class="n">client</span><span class="k">:</span> <span class="kt">Client</span><span class="o">[</span><span class="kt">Task</span><span class="o">],</span>
<span class="n">console</span><span class="k">:</span> <span class="kt">Console.Service</span><span class="o">,</span>
<span class="n">clock</span><span class="k">:</span> <span class="kt">Clock.Service</span><span class="o">,</span>
<span class="n">configuration</span><span class="k">:</span> <span class="kt">Configuration</span>
<span class="o">)</span> <span class="k">extends</span> <span class="nc">TempClient</span> <span class="o">{</span>
<span class="k">import</span> <span class="nn">org.http4s.circe._</span>
<span class="k">import</span> <span class="nn">io.circe.generic.auto._</span>
<span class="k">import</span> <span class="nn">TempClientLive.OpenWeather._</span>
<span class="k">implicit</span> <span class="k">val</span> <span class="n">userDecoder</span> <span class="k">=</span> <span class="n">jsonOf</span><span class="o">[</span><span class="kt">Task</span>, <span class="kt">OWResult</span><span class="o">]</span>
<span class="k">val</span> <span class="n">tempConfig</span> <span class="k">=</span> <span class="n">configuration</span><span class="o">.</span><span class="n">config</span><span class="o">.</span><span class="n">temperatureConfig</span>
<span class="k">override</span> <span class="k">val</span> <span class="n">temperatureStream</span><span class="k">:</span> <span class="kt">ZStream</span><span class="o">[</span><span class="kt">Any</span>, <span class="kt">Throwable</span>, <span class="kt">Temperature</span><span class="o">]</span> <span class="k">=</span>
<span class="c1">// emit one right away and then do the rest in an interval
</span> <span class="o">(</span><span class="nc">ZStream</span><span class="o">.</span><span class="n">succeed</span><span class="o">(-</span><span class="mi">1L</span><span class="o">)</span> <span class="o">++</span> <span class="nc">ZStream</span><span class="o">.</span><span class="n">fromSchedule</span><span class="o">(</span><span class="nc">Schedule</span><span class="o">.</span><span class="n">spaced</span><span class="o">(</span><span class="n">tempConfig</span><span class="o">.</span><span class="n">interval</span><span class="o">.</span><span class="n">toJava</span><span class="o">)))</span>
<span class="o">.</span><span class="n">mapM</span><span class="o">(</span><span class="k">_</span> <span class="k">=></span> <span class="n">makeTemperatureCall</span><span class="o">(</span><span class="n">tempConfig</span><span class="o">.</span><span class="n">endpoint</span> <span class="o">+</span> <span class="n">tempConfig</span><span class="o">.</span><span class="n">apiKey</span><span class="o">))</span>
<span class="o">.</span><span class="n">tap</span> <span class="o">{</span> <span class="n">p</span> <span class="k">=></span> <span class="n">console</span><span class="o">.</span><span class="n">putStrLn</span><span class="o">(</span><span class="n">p</span><span class="o">.</span><span class="n">toString</span><span class="o">())</span> <span class="o">}</span>
<span class="c1">// We still have to provide the clock for the schedule to eliminate all R
</span> <span class="c1">// and just use everything provided to this service.
</span> <span class="o">.</span><span class="n">provide</span><span class="o">(</span><span class="nc">Has</span><span class="o">(</span><span class="n">clock</span><span class="o">))</span>
<span class="cm">/** Make the call. This requires a client to be in the environment, and returns a string
*/</span>
<span class="k">private</span> <span class="k">def</span> <span class="n">makeTemperatureCall</span><span class="o">(</span><span class="n">url</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span><span class="k">:</span> <span class="kt">ZIO</span><span class="o">[</span><span class="kt">Any</span>, <span class="kt">Throwable</span>, <span class="kt">Temperature</span><span class="o">]</span> <span class="k">=</span> <span class="o">{</span>
<span class="k">for</span> <span class="o">{</span>
<span class="c1">// for converting to string, we can use the standard EntityDecoder from HTTP4S together with
</span> <span class="c1">// the zio.interop.cats_ for mapping the Cats stuff to ZIO
</span> <span class="n">res</span> <span class="k"><-</span> <span class="n">client</span><span class="o">.</span><span class="n">expect</span><span class="o">[</span><span class="kt">OWResult</span><span class="o">](</span><span class="n">url</span><span class="o">)</span>
<span class="o">}</span> <span class="k">yield</span> <span class="o">(</span><span class="nc">Temperature</span><span class="o">(</span><span class="n">res</span><span class="o">.</span><span class="n">dt</span><span class="o">,</span> <span class="n">res</span><span class="o">.</span><span class="n">main</span><span class="o">.</span><span class="n">temp</span><span class="o">))</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>The stream is exposed as a value <code class="highlighter-rouge">temperatureStream</code> when we create this service. Basically what we do is we first create a <code class="highlighter-rouge">ZStream</code> from a single value <code class="highlighter-rouge">ZStream.succeed</code>, and concat that one with the <code class="highlighter-rouge">ZStream.fromSchedule(Schedule.spaced(tempConfig.interval.toJava))</code>. The second one will run once every minute, based on the <code class="highlighter-rouge">Schedule</code> that we passed in through the <code class="highlighter-rouge">configuration</code> dependency. Once we start consuming this stream, at each <code class="highlighter-rouge">tick</code>, we use <code class="highlighter-rouge">mapM</code> to call <code class="highlighter-rouge">makeTemperatureCall</code>, which uses the <code class="highlighter-rouge">client</code> dependency to make a REST call, that returns the case class, which the consumer can then process (in our case, the consumer will store it in MongoDB). We use <code class="highlighter-rouge">MapM</code> here, since the <code class="highlighter-rouge">makeTemperatureCall</code> returns a ZIO:</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="n">mapM</span><span class="o">[</span><span class="kt">R1</span> <span class="k"><:</span> <span class="kt">R</span>, <span class="kt">E1</span> <span class="k">>:</span> <span class="kt">E</span>, <span class="kt">O2</span><span class="o">](</span><span class="n">f</span><span class="k">:</span> <span class="kt">O</span> <span class="o">=></span> <span class="nc">ZIO</span><span class="o">[</span><span class="kt">R1</span>, <span class="kt">E1</span>, <span class="kt">O2</span><span class="o">])</span><span class="k">:</span> <span class="kt">ZStream</span><span class="o">[</span><span class="kt">R1</span>, <span class="kt">E1</span>, <span class="kt">O2</span><span class="o">]</span>
</code></pre></div></div>
<p>Finally you can see that we use <code class="highlighter-rouge">provide(Has(clock))</code> on the <code class="highlighter-rouge">ZStream</code> we created. Doing that eliminates the dependency of the resulting stream on <code class="highlighter-rouge">Has[Clock.Service]</code>, and results in a value that, when called by the consumer, doesn’t require anything specific in the environment. We could have left out the provide but then the signature of this function would change:</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// with provide
</span><span class="k">val</span> <span class="n">temperatureStream</span><span class="k">:</span> <span class="kt">ZStream</span><span class="o">[</span><span class="kt">Any</span>, <span class="kt">Throwable</span>, <span class="kt">Temperature</span><span class="o">]</span>
<span class="c1">// without provide
</span><span class="k">val</span> <span class="n">temperatureStream</span><span class="k">:</span> <span class="kt">ZStream</span><span class="o">[</span><span class="kt">Has</span><span class="o">[</span><span class="kt">Clock.Service</span><span class="o">]</span>, <span class="kt">Throwable</span>, <span class="kt">Temperature</span><span class="o">]</span>
</code></pre></div></div>
<p>It’s debatable whether we should leave the dependency in the signature or remove it. For me I think it is cleaner to pass the dependency during the creation of the <code class="highlighter-rouge">Live</code> layer, since it is more implementation focussed, and doesn’t really deal with any business logic needed in the environment (e.g I <em>would</em> add a <code class="highlighter-rouge">UserContext</code> explicitly to the trait).</p>
<p>For completeness sake I’ll also show the implementation of the <code class="highlighter-rouge">Has[Client]</code> (the HTTP4S client):</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">object</span> <span class="nc">http4sClient</span> <span class="o">{</span>
<span class="k">object</span> <span class="nc">Http4sClientLive</span> <span class="o">{</span>
<span class="k">val</span> <span class="n">layer</span><span class="k">:</span> <span class="kt">ZLayer</span><span class="o">[</span><span class="kt">Any</span>, <span class="kt">Throwable</span>, <span class="kt">Has</span><span class="o">[</span><span class="kt">Client</span><span class="o">[</span><span class="kt">Task</span><span class="o">]]]</span> <span class="k">=</span> <span class="o">{</span>
<span class="k">implicit</span> <span class="k">val</span> <span class="n">runtime</span><span class="k">:</span> <span class="kt">Runtime</span><span class="o">[</span><span class="kt">ZEnv</span><span class="o">]</span> <span class="k">=</span> <span class="nc">Runtime</span><span class="o">.</span><span class="n">default</span>
<span class="k">val</span> <span class="n">res</span> <span class="k">=</span> <span class="nc">BlazeClientBuilder</span><span class="o">[</span><span class="kt">Task</span><span class="o">](</span><span class="n">runtime</span><span class="o">.</span><span class="n">platform</span><span class="o">.</span><span class="n">executor</span><span class="o">.</span><span class="n">asEC</span><span class="o">).</span><span class="n">resource</span><span class="o">.</span><span class="n">toManagedZIO</span>
<span class="nc">ZLayer</span><span class="o">.</span><span class="n">fromManaged</span><span class="o">(</span><span class="n">res</span><span class="o">)</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Here it isn’t really useful to create a trait and use construction based injection of dependencies, so we just create the implementation directly when we access the <code class="highlighter-rouge">layer</code>.</p>
<h2 id="storing-data-in-the-database">Storing data in the database</h2>
<p>Before we connect all the different parts lets look at the mongoDB stuff. To connect to Mongo we’re going to need a <code class="highlighter-rouge">MongoDBClient</code>:</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">object</span> <span class="nc">mongo</span> <span class="o">{</span>
<span class="k">object</span> <span class="nc">MongoDBConnectionLive</span> <span class="o">{</span>
<span class="cm">/** Slightly different approach for when we're using case classes, since we don't really expose
* the functions on the service, but want to expose a mongoDB connection directly.
*/</span>
<span class="k">val</span> <span class="n">managedMongoClient</span><span class="k">:</span> <span class="kt">ZManaged</span><span class="o">[</span><span class="kt">Has</span><span class="o">[</span><span class="kt">Configuration</span><span class="o">]</span>, <span class="kt">Throwable</span>, <span class="kt">MongoClient</span><span class="o">]</span> <span class="k">=</span> <span class="k">for</span> <span class="o">{</span>
<span class="n">config</span> <span class="k"><-</span> <span class="nc">Configuration</span><span class="o">.</span><span class="n">config</span><span class="o">.</span><span class="n">toManaged_</span>
<span class="n">mongoClient</span> <span class="k"><-</span> <span class="nc">ZManaged</span><span class="o">.</span><span class="n">make</span><span class="o">(</span><span class="n">acquireConnection</span><span class="o">(</span><span class="n">config</span><span class="o">.</span><span class="n">dbConfig</span><span class="o">.</span><span class="n">endpoint</span><span class="o">))(</span><span class="n">releaseConnection</span><span class="o">(</span><span class="k">_</span><span class="o">))</span>
<span class="o">}</span> <span class="k">yield</span> <span class="o">(</span><span class="n">mongoClient</span><span class="o">)</span>
<span class="cm">/** Try and connect the database
*/</span>
<span class="k">private</span> <span class="k">def</span> <span class="n">acquireConnection</span><span class="o">(</span><span class="n">databaseUri</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span><span class="k">:</span> <span class="kt">Task</span><span class="o">[</span><span class="kt">MongoClient</span><span class="o">]</span> <span class="k">=</span> <span class="nc">ZIO</span><span class="o">.</span><span class="n">fromTry</span><span class="o">(</span><span class="nc">Try</span> <span class="o">{</span>
<span class="nc">MongoClient</span><span class="o">(</span><span class="n">databaseUri</span><span class="o">)</span>
<span class="o">})</span>
<span class="cm">/** Release the connection. If an error occurs during releasing, we just ignore it
* for now. It would probably be better to check the error, whether it can be ignored
* and log some stuff. But for now this should be enough
*
* @param mongoClient the client for which we want to release the connection
* @return effect that will always succeed
*/</span>
<span class="k">private</span> <span class="k">def</span> <span class="n">releaseConnection</span><span class="o">(</span><span class="n">mongoClient</span><span class="k">:</span> <span class="kt">MongoClient</span><span class="o">)</span><span class="k">:</span> <span class="kt">ZIO</span><span class="o">[</span><span class="kt">Any</span>, <span class="kt">Nothing</span>, <span class="kt">Unit</span><span class="o">]</span> <span class="k">=</span>
<span class="nc">ZIO</span><span class="o">.</span><span class="n">fromTry</span><span class="o">(</span><span class="nc">Try</span><span class="o">(</span><span class="n">mongoClient</span><span class="o">.</span><span class="n">close</span><span class="o">)).</span><span class="n">orElse</span><span class="o">(</span><span class="nc">ZIO</span><span class="o">.</span><span class="n">succeed</span><span class="o">())</span>
<span class="cm">/** For this component, we're not going to create a case class, since the construction of this layer
* can fail, and we've got a managed resource for which we want to create a connection.
*/</span>
<span class="k">val</span> <span class="n">layer</span><span class="k">:</span> <span class="kt">ZLayer</span><span class="o">[</span><span class="kt">Has</span><span class="o">[</span><span class="kt">Configuration</span><span class="o">]</span>, <span class="kt">Throwable</span>, <span class="kt">Has</span><span class="o">[</span><span class="kt">MongoClient</span><span class="o">]]</span> <span class="k">=</span> <span class="nc">ZLayer</span><span class="o">.</span><span class="n">fromManaged</span><span class="o">(</span><span class="n">managedMongoClient</span><span class="o">)</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>The code above should look rather familiar by now. We create a <code class="highlighter-rouge">ZLayer</code> from a managed resource. So we specify how to acquire a connection, and what to do when we release it. For the rest nothing that exiting. Once we have the above wrapped resource, we can use it in our repository implementation.</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">object</span> <span class="nc">storage</span> <span class="o">{</span>
<span class="cm">/** The trait for storing.
*/</span>
<span class="k">trait</span> <span class="nc">TemperatureStorage</span> <span class="o">{</span>
<span class="k">def</span> <span class="n">insert</span><span class="o">(</span><span class="n">temperature</span><span class="k">:</span> <span class="kt">Temperature</span><span class="o">)</span><span class="k">:</span> <span class="kt">ZIO</span><span class="o">[</span><span class="kt">Any</span>, <span class="kt">Throwable</span>, <span class="kt">Unit</span><span class="o">]</span>
<span class="k">def</span> <span class="n">getAll</span><span class="o">()</span><span class="k">:</span> <span class="kt">ZIO</span><span class="o">[</span><span class="kt">Any</span>, <span class="kt">Throwable</span>, <span class="kt">List</span><span class="o">[</span><span class="kt">Temperature</span><span class="o">]]</span>
<span class="o">}</span>
<span class="cm">/** The implementation of temp storage, very naive for now, just to show the different parts connected to one another
*
* @param configuration
*/</span>
<span class="k">case</span> <span class="k">class</span> <span class="nc">TemperatureStorageLive</span><span class="o">(</span><span class="n">configuration</span><span class="k">:</span> <span class="kt">Configuration</span><span class="o">,</span> <span class="n">mongoClient</span><span class="k">:</span> <span class="kt">MongoClient</span><span class="o">)</span> <span class="k">extends</span> <span class="nc">TemperatureStorage</span> <span class="o">{</span>
<span class="k">val</span> <span class="n">temperatureCodecProvider</span> <span class="k">=</span> <span class="nc">Macros</span><span class="o">.</span><span class="n">createCodecProvider</span><span class="o">[</span><span class="kt">Temperature</span><span class="o">]()</span>
<span class="k">val</span> <span class="n">codecRegistry</span> <span class="k">=</span> <span class="n">fromRegistries</span><span class="o">(</span><span class="n">fromProviders</span><span class="o">(</span><span class="n">temperatureCodecProvider</span><span class="o">),</span> <span class="nc">DEFAULT_CODEC_REGISTRY</span><span class="o">)</span>
<span class="cm">/** Try and store the temperature string
* @param temp element to store
* @return ZIO containing the result
*/</span>
<span class="k">override</span> <span class="k">def</span> <span class="n">insert</span><span class="o">(</span><span class="n">temperature</span><span class="k">:</span> <span class="kt">Temperature</span><span class="o">)</span><span class="k">:</span> <span class="kt">ZIO</span><span class="o">[</span><span class="kt">Any</span>, <span class="kt">Throwable</span>, <span class="kt">Unit</span><span class="o">]</span> <span class="k">=</span> <span class="n">withCollection</span><span class="o">[</span><span class="kt">Unit</span>, <span class="kt">Temperature</span><span class="o">]</span> <span class="o">{</span>
<span class="n">collection</span> <span class="k">=></span>
<span class="n">collection</span>
<span class="c1">// insert the entry
</span> <span class="o">.</span><span class="n">insertOne</span><span class="o">(</span><span class="n">temperature</span><span class="o">)</span>
<span class="c1">// convert to a single result, since the result is a singleObservable
</span> <span class="o">.</span><span class="n">toStream</span><span class="o">()</span>
<span class="o">.</span><span class="n">runHead</span>
<span class="c1">// we should do some more error handling here in a real world scenario
</span> <span class="o">.</span><span class="n">flatMap</span> <span class="o">{</span>
<span class="k">case</span> <span class="nc">Some</span><span class="o">(</span><span class="n">res</span><span class="o">)</span> <span class="k">=></span> <span class="nc">ZIO</span><span class="o">.</span><span class="n">succeed</span><span class="o">()</span>
<span class="k">case</span> <span class="nc">None</span> <span class="k">=></span> <span class="nc">ZIO</span><span class="o">.</span><span class="n">fail</span><span class="o">(</span><span class="k">new</span> <span class="nc">IllegalArgumentException</span><span class="o">(</span><span class="s">"Expected result from mongodb"</span><span class="o">))</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="cm">/** Return all the elements we currently have
*
* @return list of all the temperatures we've got stored
*/</span>
<span class="k">override</span> <span class="k">def</span> <span class="n">getAll</span><span class="o">()</span><span class="k">:</span> <span class="kt">ZIO</span><span class="o">[</span><span class="kt">Any</span>, <span class="kt">Throwable</span>, <span class="kt">List</span><span class="o">[</span><span class="kt">Temperature</span><span class="o">]]</span> <span class="k">=</span> <span class="n">withCollection</span><span class="o">[</span><span class="kt">List</span><span class="o">[</span><span class="kt">Temperature</span><span class="o">]</span>, <span class="kt">Temperature</span><span class="o">]</span> <span class="o">{</span>
<span class="k">_</span><span class="o">.</span><span class="n">find</span><span class="o">()</span>
<span class="o">.</span><span class="n">toStream</span><span class="o">()</span>
<span class="o">.</span><span class="n">runCollect</span>
<span class="o">.</span><span class="n">map</span><span class="o">(</span><span class="k">_</span><span class="o">.</span><span class="n">toList</span><span class="o">)</span>
<span class="o">}</span>
<span class="cm">/** Get the database and collection to which to store.
*
* @param f function to call within the context of this collection
* @return result of wrapped
*/</span>
<span class="k">private</span> <span class="k">def</span> <span class="n">withCollection</span><span class="o">[</span><span class="kt">A</span>, <span class="kt">T:</span> <span class="kt">ClassTag</span><span class="o">](</span>
<span class="n">f</span><span class="k">:</span> <span class="kt">MongoCollection</span><span class="o">[</span><span class="kt">T</span><span class="o">]</span> <span class="k">=></span> <span class="nc">ZIO</span><span class="o">[</span><span class="kt">Any</span>, <span class="kt">Throwable</span>, <span class="kt">A</span><span class="o">]</span>
<span class="o">)</span><span class="k">:</span> <span class="kt">ZIO</span><span class="o">[</span><span class="kt">Any</span>, <span class="kt">Throwable</span>, <span class="kt">A</span><span class="o">]</span> <span class="k">=</span> <span class="o">{</span>
<span class="k">val</span> <span class="n">collectionZIO</span> <span class="k">=</span> <span class="nc">ZIO</span><span class="o">.</span><span class="n">fromTry</span><span class="o">(</span><span class="nc">Try</span> <span class="o">{</span>
<span class="c1">// TODO: we should get the database name at least from the configuration
</span> <span class="n">mongoClient</span><span class="o">.</span><span class="n">getDatabase</span><span class="o">(</span><span class="s">"sampleservice"</span><span class="o">).</span><span class="n">withCodecRegistry</span><span class="o">(</span><span class="n">codecRegistry</span><span class="o">).</span><span class="n">getCollection</span><span class="o">[</span><span class="kt">T</span><span class="o">](</span><span class="s">"temperatures"</span><span class="o">)</span>
<span class="o">})</span>
<span class="n">collectionZIO</span><span class="o">.</span><span class="n">flatMap</span><span class="o">(</span><span class="n">f</span><span class="o">)</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="k">object</span> <span class="nc">TemperatureStorageLive</span> <span class="o">{</span>
<span class="k">val</span> <span class="n">layer</span><span class="k">:</span> <span class="kt">URLayer</span><span class="o">[</span><span class="kt">Has</span><span class="o">[</span><span class="kt">Configuration</span><span class="o">]</span> <span class="kt">with</span> <span class="kt">Has</span><span class="o">[</span><span class="kt">MongoClient</span><span class="o">]</span>, <span class="kt">Has</span><span class="o">[</span><span class="kt">TemperatureStorage</span><span class="o">]]</span> <span class="k">=</span>
<span class="o">(</span><span class="nc">TemperatureStorageLive</span><span class="o">(</span><span class="k">_</span><span class="o">,</span> <span class="k">_</span><span class="o">)).</span><span class="n">toLayer</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p><strong>Disclaimer:</strong> There are already other MongoDB clients out there that use ZIO, I just created one from scratch to show how easy it really is to interop with existing libraries and tools.</p>
<p>The code above follows the same module pattern, where we define a trait, a case class implementing the trait, and pass in any required dependencies by using the <code class="highlighter-rouge">toLayer</code> function. Most of the code above is just making sure we can work with MongoDB, and the only interesting stuff is happening in the <code class="highlighter-rouge">insert</code> and <code class="highlighter-rouge">getAll</code> functions.</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">override</span> <span class="k">def</span> <span class="n">insert</span><span class="o">(</span><span class="n">temperature</span><span class="k">:</span> <span class="kt">Temperature</span><span class="o">)</span><span class="k">:</span> <span class="kt">ZIO</span><span class="o">[</span><span class="kt">Any</span>, <span class="kt">Throwable</span>, <span class="kt">Unit</span><span class="o">]</span> <span class="k">=</span> <span class="n">withCollection</span><span class="o">[</span><span class="kt">Unit</span>, <span class="kt">Temperature</span><span class="o">]</span> <span class="o">{</span>
<span class="n">collection</span> <span class="k">=></span>
<span class="n">collection</span>
<span class="c1">// insert the entry
</span> <span class="o">.</span><span class="n">insertOne</span><span class="o">(</span><span class="n">temperature</span><span class="o">)</span>
<span class="c1">// convert to a single result, since the result is a singleObservable
</span> <span class="o">.</span><span class="n">toStream</span><span class="o">()</span>
<span class="o">.</span><span class="n">runHead</span>
<span class="c1">// we should do some more error handling here in a real world scenario
</span> <span class="o">.</span><span class="n">flatMap</span> <span class="o">{</span>
<span class="k">case</span> <span class="nc">Some</span><span class="o">(</span><span class="n">res</span><span class="o">)</span> <span class="k">=></span> <span class="nc">ZIO</span><span class="o">.</span><span class="n">succeed</span><span class="o">()</span>
<span class="k">case</span> <span class="nc">None</span> <span class="k">=></span> <span class="nc">ZIO</span><span class="o">.</span><span class="n">fail</span><span class="o">(</span><span class="k">new</span> <span class="nc">IllegalArgumentException</span><span class="o">(</span><span class="s">"Expected result from mongodb"</span><span class="o">))</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>When you look at the <code class="highlighter-rouge">insertOne</code> signature:</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">def</span> <span class="n">insertOne</span><span class="o">(</span><span class="n">document</span><span class="k">:</span> <span class="kt">TResult</span><span class="o">)</span><span class="k">:</span> <span class="kt">SingleObservable</span><span class="o">[</span><span class="kt">InsertOneResult</span><span class="o">]</span> <span class="k">=</span> <span class="n">wrapped</span><span class="o">.</span><span class="n">insertOne</span><span class="o">(</span><span class="n">document</span><span class="o">)</span>
</code></pre></div></div>
<p>You’ll see that it returns a <code class="highlighter-rouge">SingleObservable</code>. This type is a trait defined like this:</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">trait</span> <span class="nc">SingleObservable</span><span class="o">[</span><span class="kt">T</span><span class="o">]</span> <span class="nc">extends</span> <span class="nc">Observable</span><span class="o">[</span><span class="kt">T</span><span class="o">]</span> <span class="o">{</span> <span class="o">...</span> <span class="o">}</span>
<span class="k">trait</span> <span class="nc">Observable</span><span class="o">[</span><span class="kt">T</span><span class="o">]</span> <span class="nc">extends</span> <span class="nc">Publisher</span><span class="o">[</span><span class="kt">T</span><span class="o">]</span> <span class="o">{</span> <span class="o">...</span> <span class="o">}</span>
</code></pre></div></div>
<p><code class="highlighter-rouge">Publisher</code> is part of the <code class="highlighter-rouge">org.reactivestreams</code> library and provides a generic interface for integration different kinds of streams. This means that we can easily convert the result from the <code class="highlighter-rouge">insertOne</code> call to a <code class="highlighter-rouge">ZStream</code> by just calling <code class="highlighter-rouge">toStream()</code> (which is a function provided by ZIO). The result is that we’ve got a <code class="highlighter-rouge">ZStream</code> which will always just return a single element. To get this element we call the <code class="highlighter-rouge">runHead</code> function, which will return a <code class="highlighter-rouge">ZIO[Option[T]]</code>, which we process in the normal way. And that’s already it, without any complex integration we can easily reuse existing libraries.</p>
<p>The <code class="highlighter-rouge">getAll</code> function is pretty much the same:</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">override</span> <span class="k">def</span> <span class="n">getAll</span><span class="o">()</span><span class="k">:</span> <span class="kt">ZIO</span><span class="o">[</span><span class="kt">Any</span>, <span class="kt">Throwable</span>, <span class="kt">List</span><span class="o">[</span><span class="kt">Temperature</span><span class="o">]]</span> <span class="k">=</span> <span class="n">withCollection</span><span class="o">[</span><span class="kt">List</span><span class="o">[</span><span class="kt">Temperature</span><span class="o">]</span>, <span class="kt">Temperature</span><span class="o">]</span> <span class="o">{</span>
<span class="k">_</span><span class="o">.</span><span class="n">find</span><span class="o">()</span>
<span class="o">.</span><span class="n">toStream</span><span class="o">()</span>
<span class="o">.</span><span class="n">runCollect</span>
<span class="o">.</span><span class="n">map</span><span class="o">(</span><span class="k">_</span><span class="o">.</span><span class="n">toList</span><span class="o">)</span>
</code></pre></div></div>
<p>Here we convert the resulting stream from the Mongo driver to a <code class="highlighter-rouge">ZStream</code>, collect all the elements in the stream using <code class="highlighter-rouge">runCollect</code>, and finally map the <code class="highlighter-rouge">Chunk[T]</code> which we get back from the <code class="highlighter-rouge">runCollect</code> to a <code class="highlighter-rouge">List[T]</code>. I was really impressed with how easy working with ZStream is, the interoperability with existing libraries, and the interaction between the ZIO types and the ZStream types.</p>
<h2 id="the-rest-endpoint-to-get-all-the-data">The rest endpoint to get all the data</h2>
<p>So we’ve covered pretty much everything I wanted to explain in this article. The only thing left is the REST route to access the stored data. For this we first define some routes:</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">object</span> <span class="nc">Routes</span> <span class="o">{</span>
<span class="k">trait</span> <span class="nc">TemperatureRoute</span> <span class="o">{</span>
<span class="k">val</span> <span class="n">routes</span><span class="k">:</span> <span class="kt">HttpRoutes</span><span class="o">[</span><span class="kt">Task</span><span class="o">]</span>
<span class="o">}</span>
<span class="k">object</span> <span class="nc">TemperatureRoute</span> <span class="o">{</span>
<span class="c1">// helper function which access the correct resource from our environment, and lifts
</span> <span class="c1">// it in an effect.
</span> <span class="k">val</span> <span class="n">temperatureRoutes</span><span class="k">:</span> <span class="kt">URIO</span><span class="o">[</span><span class="kt">Has</span><span class="o">[</span><span class="kt">TemperatureRoute</span><span class="o">]</span>, <span class="kt">HttpRoutes</span><span class="o">[</span><span class="kt">Task</span><span class="o">]]</span> <span class="k">=</span> <span class="nc">ZIO</span><span class="o">.</span><span class="n">access</span><span class="o">(</span><span class="k">_</span><span class="o">.</span><span class="n">get</span><span class="o">.</span><span class="n">routes</span><span class="o">)</span>
<span class="o">}</span>
<span class="k">object</span> <span class="nc">TemperatureRouteLive</span> <span class="o">{</span>
<span class="k">private</span> <span class="k">val</span> <span class="n">dsl</span> <span class="k">=</span> <span class="nc">Http4sDsl</span><span class="o">[</span><span class="kt">Task</span><span class="o">]</span>
<span class="k">import</span> <span class="nn">dsl._</span>
<span class="k">import</span> <span class="nn">io.circe.generic.auto._</span><span class="o">,</span> <span class="n">io</span><span class="o">.</span><span class="n">circe</span><span class="o">.</span><span class="n">syntax</span><span class="o">.</span><span class="k">_</span>
<span class="k">import</span> <span class="nn">org.http4s.dsl.Http4sDsl</span>
<span class="k">import</span> <span class="nn">org.http4s.circe._</span>
<span class="k">import</span> <span class="nn">zio.interop.catz._</span>
<span class="cm">/** A simple layer which returns the routes for the temperature.
*/</span>
<span class="k">val</span> <span class="n">layer</span><span class="k">:</span> <span class="kt">ZLayer</span><span class="o">[</span><span class="kt">Has</span><span class="o">[</span><span class="kt">storage.TemperatureStorage</span><span class="o">]</span>, <span class="kt">Nothing</span>, <span class="kt">Has</span><span class="o">[</span><span class="kt">TemperatureRoute</span><span class="o">]]</span> <span class="k">=</span>
<span class="nc">ZLayer</span><span class="o">.</span><span class="n">fromService</span> <span class="o">{</span> <span class="n">storage</span> <span class="k">=></span>
<span class="k">new</span> <span class="nc">TemperatureRoute</span> <span class="o">{</span>
<span class="k">val</span> <span class="n">routes</span> <span class="k">=</span> <span class="nc">HttpRoutes</span>
<span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">Task</span><span class="o">]</span> <span class="o">{</span>
<span class="k">case</span> <span class="nc">GET</span> <span class="o">-></span> <span class="nc">Root</span> <span class="o">/</span> <span class="s">"temperatures"</span> <span class="k">=></span> <span class="o">{</span>
<span class="n">storage</span><span class="o">.</span><span class="n">getAll</span><span class="o">().</span><span class="n">flatMap</span> <span class="o">{</span> <span class="n">all</span> <span class="k">=></span> <span class="nc">Ok</span><span class="o">(</span><span class="n">all</span><span class="o">.</span><span class="n">asJson</span><span class="o">)</span> <span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>As you can see I defined these as a separate layer, which requires access to the database to get the list of the stored data. This way we can just define the routes in isolation, and to add them we just mark them as dependency for the HTTP4S server.</p>
<p>With all this out of the way, we can start the server and make an HTTP call to get the (very boring looking) data.</p>
<h2 id="combining-everything">combining everything</h2>
<p>To start this, we first setup the layers and then define a simple program:</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">object</span> <span class="nc">Application</span> <span class="k">extends</span> <span class="n">zio</span><span class="o">.</span><span class="nc">App</span> <span class="o">{</span>
<span class="c1">// combine the configuration layer and the standard environment into
</span> <span class="c1">// a new layer.
</span> <span class="k">val</span> <span class="n">defaultLayer</span> <span class="k">=</span> <span class="nc">ConfigurationLive</span><span class="o">.</span><span class="n">layer</span> <span class="o">++</span> <span class="nc">ZEnv</span><span class="o">.</span><span class="n">live</span>
<span class="c1">// the storage layer
</span> <span class="k">val</span> <span class="n">storageLayer</span> <span class="k">=</span> <span class="n">defaultLayer</span> <span class="o">>+></span> <span class="n">db</span><span class="o">.</span><span class="n">mongo</span><span class="o">.</span><span class="nc">MongoDBConnectionLive</span><span class="o">.</span><span class="n">layer</span> <span class="o">>>></span> <span class="n">storage</span><span class="o">.</span><span class="nc">TemperatureStorageLive</span><span class="o">.</span><span class="n">layer</span>
<span class="k">val</span> <span class="n">httpsLayer</span> <span class="k">=</span>
<span class="n">storageLayer</span> <span class="o">>+></span> <span class="nc">TemperatureRouteLive</span><span class="o">.</span><span class="n">layer</span> <span class="o">++</span> <span class="n">defaultLayer</span> <span class="o">>+></span> <span class="n">http4sServer</span><span class="o">.</span><span class="nc">Http4sServerLive</span><span class="o">.</span><span class="n">layer</span> <span class="o">++</span> <span class="n">http4sClient</span><span class="o">.</span><span class="nc">Http4sClientLive</span><span class="o">.</span><span class="n">layer</span>
<span class="c1">// combine the layers into a single layer to feed into the program
</span> <span class="k">val</span> <span class="n">applicationLayer</span> <span class="k">=</span>
<span class="n">httpsLayer</span> <span class="o">>+></span> <span class="n">tempClient</span><span class="o">.</span><span class="nc">TempClientLive</span><span class="o">.</span><span class="n">layer</span> <span class="o">++</span> <span class="n">storageLayer</span>
<span class="cm">/** Provide the layer to the program and run it
*
* @param args
* @return
*/</span>
<span class="k">override</span> <span class="k">def</span> <span class="n">run</span><span class="o">(</span><span class="n">args</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">String</span><span class="o">])</span><span class="k">:</span> <span class="kt">URIO</span><span class="o">[</span><span class="kt">ZEnv</span>, <span class="kt">ExitCode</span><span class="o">]</span> <span class="k">=</span>
<span class="n">pp</span><span class="o">.</span><span class="n">provideLayer</span><span class="o">(</span><span class="n">applicationLayer</span><span class="o">).</span><span class="n">exitCode</span>
<span class="cm">/** Helper type to indicate which dependencies are needed to run the program
*/</span>
<span class="k">type</span> <span class="kt">ProgramDeps</span> <span class="o">=</span> <span class="nc">Has</span><span class="o">[</span><span class="kt">Server</span><span class="o">]</span>
<span class="k">with</span> <span class="nc">Has</span><span class="o">[</span><span class="kt">storage.TemperatureStorage</span><span class="o">]</span>
<span class="k">with</span> <span class="nc">Has</span><span class="o">[</span><span class="kt">tempClient.TempClient</span><span class="o">]</span>
<span class="k">with</span> <span class="nc">Has</span><span class="o">[</span><span class="kt">Client</span><span class="o">[</span><span class="kt">Task</span><span class="o">]]</span>
<span class="k">with</span> <span class="n">zio</span><span class="o">.</span><span class="n">clock</span><span class="o">.</span><span class="nc">Clock</span>
<span class="k">with</span> <span class="n">zio</span><span class="o">.</span><span class="n">console</span><span class="o">.</span><span class="nc">Console</span>
<span class="cm">/** The program that starts polling the https endpoint and write the results to
* the database.
*/</span>
<span class="k">val</span> <span class="n">pp</span><span class="k">:</span> <span class="kt">ZIO</span><span class="o">[</span><span class="kt">ProgramDeps</span>, <span class="kt">Throwable</span>, <span class="kt">Unit</span><span class="o">]</span> <span class="k">=</span> <span class="o">{</span>
<span class="k">for</span> <span class="o">{</span>
<span class="c1">// start the stream, which polls the temperature endpoint
</span> <span class="n">stream</span> <span class="k"><-</span> <span class="n">tempClient</span><span class="o">.</span><span class="nc">TempClient</span><span class="o">.</span><span class="n">temperatureStream</span>
<span class="n">mapped</span> <span class="k">=</span> <span class="n">stream</span><span class="o">.</span><span class="n">mapM</span><span class="o">(</span><span class="n">temperatureString</span> <span class="k">=></span>
<span class="nc">ZIO</span><span class="o">.</span><span class="n">serviceWith</span><span class="o">[</span><span class="kt">storage.TemperatureStorage</span><span class="o">](</span><span class="k">_</span><span class="o">.</span><span class="n">insert</span><span class="o">(</span><span class="n">temperatureString</span><span class="o">))</span>
<span class="o">)</span>
<span class="k">_</span> <span class="k"><-</span> <span class="n">mapped</span><span class="o">.</span><span class="n">runDrain</span>
<span class="o">}</span> <span class="k">yield</span> <span class="o">()</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>In this program the only thing we do is, that we get the stream of temperature updates (<code class="highlighter-rouge">tempClient.TempClient.temperatureStream</code>), and for each element in the stream we define that we want to store it in database using the <code class="highlighter-rouge">mapM</code> function. We run this stream by calling <code class="highlighter-rouge">runDrain</code>, which will just keep waiting for new elements in the stream and process those.</p>
<p>When we run this, we see output like this:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[zio-default-async-1] INFO org.mongodb.driver.cluster - Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms'}
[cluster-ClusterId{value='60ad3ce890de0d65166e20ab', description='null'}-localhost:27017] INFO org.mongodb.driver.connection - Opened connection [connectionId{localValue:2, serverValue:7}] to localhost:27017
[cluster-ClusterId{value='60ad3ce890de0d65166e20ab', description='null'}-localhost:27017] INFO org.mongodb.driver.cluster - Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=8, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=18662711}
[cluster-rtt-ClusterId{value='60ad3ce890de0d65166e20ab', description='null'}-localhost:27017] INFO org.mongodb.driver.connection - Opened connection [connectionId{localValue:1, serverValue:8}] to localhost:27017
[zio-default-async-5] INFO org.http4s.blaze.channel.nio1.NIO1SocketServerGroup - Service bound to address /127.0.0.1:8081
[zio-default-async-5] INFO org.http4s.server.blaze.BlazeServerBuilder -
_ _ _ _ _
| |_| |_| |_ _ __| | | ___
| ' \ _| _| '_ \_ _(_-<
|_||_\__|\__| .__/ |_|/__/
|_|
[zio-default-async-5] INFO org.http4s.server.blaze.BlazeServerBuilder - http4s v1.0.0-M4 on blaze v0.14.13 started at http://127.0.0.1:8081/
Temperature(1621966031,285.05)
[InnocuousThread-2] INFO org.mongodb.driver.connection - Opened connection [connectionId{localValue:3, serverValue:9}] to localhost:27017
</code></pre></div></div>
<p>And when we call into the webservice the result is something like this:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ curl -v localhost:8081/temperatures
* Trying 127.0.0.1:8081...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8081 (#0)
> GET /temperatures HTTP/1.1
> Host: localhost:8081
> User-Agent: curl/7.68.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Content-Type: application/json
< Date: Tue, 25 May 2021 18:08:54 GMT
< Content-Length: 780
<
* Connection #0 to host localhost left intact
[{"timestamp":1621876805,"temperature":284.12},{"timestamp":1621876918,"temperature":284.1},{"timestamp":1621876805,"temperature":284.12},{"timestamp":1621876805,"temperature":284.12},{"timestamp":1621876918,"temperature":284.1},{"timestamp":1621876918,"temperature":284.1},{"timestamp":1621877523,"temperature":283.79},{"timestamp":1621877523,"temperature":283.79},{"timestamp":1621877644,"temperature":283.76},{"timestamp":1621877644,"temperature":283.76},{"timestamp":1621877644,"temperature":283.76},{"timestamp":1621877644,"temperature":283.76},{"timestamp":1621965947,"temperature":285.05},{"timestamp":1621965947,"temperature":285.05},{"timestamp":1621966031,"temperature":285.05},{"timestamp":1621966031,"temperature":285.05},{"timestamp":1621966031,"temperature":285.05}]
</code></pre></div></div>Jos DirksenIn the previous article we looked at the basics of ZIO. In this part we’re going to extend on that example and add the following: ZStream integration: we’re going to create our own stream that polls an external service, and we’re going to use the reactive-stream integration to query a Mongo database. Storage layer: we’ll add persistence to the example, where everything we get from the external service is stored in a database (MongoDB in our case). Module pattern 2.0: we’ve rewritten a couple of services to the new module pattern. Not all services could be rewritten though, but I’ll try to explain my reasoning why and why not. Real REST endpoint: We’ll connect the HTTP4S endpoint to the MongoDB server to actually retrieve some data. We’ll not look at all the code (which you can find here: https://github.com/josdirksen/zio-playground), but focus on the main data flow and how that is connected. The main steps we’ll explore in this article are: Poll an external REST endpoint to retrieve temperature data. This data is then stored in MongoDB. Expose a REST endpoint to query all the stored data. Getting started First of, let’s quickly look at the sbt dependencies so you get a sense of which libraries we’ll be using: import Dependencies._ ThisBuild / scalaVersion := "2.13.4" ThisBuild / version := "0.1.0-SNAPSHOT" ThisBuild / organization := "org.smartjava" ThisBuild / organizationName := "smartjava" // need to use an older version, since the newest version of http4s // supports the latest cats version, while the zio-interop-cats one // doesn't support this yet. val Http4sVersion = "1.0.0-M4" val ZioVersion = "1.0.8" lazy val root = (project in file(".")) .settings( name := "zio-playground", libraryDependencies += scalaTest % Test, libraryDependencies ++= Seq( "org.http4s" %% "http4s-blaze-server" % Http4sVersion, "org.http4s" %% "http4s-dsl" % Http4sVersion, "org.http4s" %% "http4s-blaze-client" % Http4sVersion, "org.http4s" %% "http4s-circe" % Http4sVersion, // JSON Mapping "io.circe" %% "circe-generic" % "0.12.3", "io.circe" %% "circe-literal" % "0.12.3", // ZIO stuff "dev.zio" %% "zio" % ZioVersion, "dev.zio" %% "zio-streams" % ZioVersion, "dev.zio" %% "zio-interop-reactivestreams" % "1.3.5", "dev.zio" %% "zio-interop-cats" % "2.4.0.0", "com.github.pureconfig" %% "pureconfig" % "0.15.0", "org.slf4j" % "slf4j-api" % "1.7.5", "org.slf4j" % "slf4j-simple" % "1.7.5", "dev.zio" %% "zio-test" % ZioVersion % "test", "dev.zio" %% "zio-test-sbt" % ZioVersion % "test", "org.mongodb.scala" %% "mongo-scala-driver" % "4.2.3" ), testFrameworks += new TestFramework("zio.test.sbt.ZTestFramework") ) Nothing to special. We use circe for the JSON stuff, use the standard mongo-scala-driver for accessing the database, and we’ve addedd the relevant zio-streams dependencies to create streams and to interact with reactivestreams. Creating our first stream We’re first going to create our own stream. This stream will be a stream of Temperature values retrieved from an external API. This is a stream that’ll poll the external service every so often, and pass on these values to anyone that is interested. All the stuff happens in the TempClient.scala file (we’ll explain the interesting parts below this code fragment): /** The temperature client calls a remote webservice ever so often and gives * access to a stream of temperatures. */ object tempClient { trait TempClient { val temperatureStream: ZStream[Any, Throwable, Temperature] } object TempClient { /** get the zstream within the context of the provided environment. This will * return a stream that, when drained, will provide a temperature update every * tick. The only dependency here is the TempClient, retrieving this stream * won't result in any errors. Note that we lift the temperatureStream in * the UIO, else we can't use the serviceWith function, which expects the * result to be A wrapped in a URIO */ val temperatureStream: URIO[Has[TempClient], ZStream[Any, Throwable, Temperature]] = ZIO.serviceWith[TempClient](s => UIO.succeed(s.temperatureStream)) } case class TempClientLive( client: Client[Task], console: Console.Service, clock: Clock.Service, configuration: Configuration ) extends TempClient { import org.http4s.circe._ import io.circe.generic.auto._ import TempClientLive.OpenWeather._ implicit val userDecoder = jsonOf[Task, OWResult] val tempConfig = configuration.config.temperatureConfig override val temperatureStream: ZStream[Any, Throwable, Temperature] = // emit one right away and then do the rest in an interval (ZStream.succeed(-1L) ++ ZStream.fromSchedule(Schedule.spaced(tempConfig.interval.toJava))) .mapM(_ => makeTemperatureCall(tempConfig.endpoint + tempConfig.apiKey)) .tap { p => console.putStrLn(p.toString()) } // We still have to provide the clock for the schedule to eliminate all R // and just use everything provided to this service. .provide(Has(clock)) /** Make the call. This requires a client to be in the environment, and returns a string */ private def makeTemperatureCall(url: String): ZIO[Any, Throwable, Temperature] = { for { // for converting to string, we can use the standard EntityDecoder from HTTP4S together with // the zio.interop.cats_ for mapping the Cats stuff to ZIO res <- client.expect[OWResult](url) } yield (Temperature(res.dt, res.main.temp)) } } object TempClientLive { object OpenWeather { // the openweather model case class OWResult(coord: OWCoord, main: OWMain, visibility: Integer, wind: OWWind, dt: Long) case class OWCoord(lat: Double, lon: Double) case class OWMain( temp: Double, feels_like: Double, temp_min: Double, temp_max: Double, pressure: Int, humidity: Int ) case class OWWind(speed: Double, deg: Long, gust: Double) } // the dependencies for this service type TempClientLiveDeps = Has[Client[Task]] with Has[Console.Service] with Has[Clock.Service] with Has[Configuration] // the layer that can be fed to other services, and which specifies what is needed by this layer val layer: URLayer[TempClientLiveDeps, Has[TempClient]] = (TempClientLive(_, _, _, _)).toLayer } } Let’s start by looking at the structure. This setup uses the Module pattern 2.0: object tempClient { trait TempClient { val temperatureStream: ZStream[Any, Throwable, Temperature] } object TempClient { val temperatureStream: URIO[Has[TempClient], ZStream[Any, Throwable, Temperature]] = ZIO.serviceWith[TempClient](s => UIO.succeed(s.temperatureStream)) } case class TempClientLive( client: Client[Task], console: Console.Service, clock: Clock.Service, configuration: Configuration ) extends TempClient { ... } object TempClientLive { ... // the dependencies for this service type TempClientLiveDeps = Has[Client[Task]] with Has[Console.Service] with Has[Clock.Service] with Has[Configuration] // the layer that can be fed to other services, and which specifies what is needed by this layer val layer: URLayer[TempClientLiveDeps, Has[TempClient]] = (TempClientLive(_, _, _, _)).toLayer } } Follow the previous link for a detailled explanation of this pattern, but the main thing that changed is that we don’t introduce a type alias for the Has[A] part, but just define our module as a simple trait trait TempClient, and implementations of this trait TempClientLive are exposed as layers. In our example you can see that our implementation has dependencies to four other modules (Has[Client[Task], Has[Console.Service], Has[Clock.Service], and Has[Configuration]). You’ll also notice the change here, that we don’t refer types anymore, but explicitly reference the Has[A] type. I like this, since it makes it much more clear that we’re talking about layers and dependencies, and not actual implementations or traits. Since our implementation now is a simple case class we can use the toLayer function to automatically inject the dependencies our service needs when creating the layer. We’ve already looked at the layers in the previous article, so the only interesting part that is left here is the creation of the stream in the TempClientLive implementation: case class TempClientLive( client: Client[Task], console: Console.Service, clock: Clock.Service, configuration: Configuration ) extends TempClient { import org.http4s.circe._ import io.circe.generic.auto._ import TempClientLive.OpenWeather._ implicit val userDecoder = jsonOf[Task, OWResult] val tempConfig = configuration.config.temperatureConfig override val temperatureStream: ZStream[Any, Throwable, Temperature] = // emit one right away and then do the rest in an interval (ZStream.succeed(-1L) ++ ZStream.fromSchedule(Schedule.spaced(tempConfig.interval.toJava))) .mapM(_ => makeTemperatureCall(tempConfig.endpoint + tempConfig.apiKey)) .tap { p => console.putStrLn(p.toString()) } // We still have to provide the clock for the schedule to eliminate all R // and just use everything provided to this service. .provide(Has(clock)) /** Make the call. This requires a client to be in the environment, and returns a string */ private def makeTemperatureCall(url: String): ZIO[Any, Throwable, Temperature] = { for { // for converting to string, we can use the standard EntityDecoder from HTTP4S together with // the zio.interop.cats_ for mapping the Cats stuff to ZIO res <- client.expect[OWResult](url) } yield (Temperature(res.dt, res.main.temp)) } } The stream is exposed as a value temperatureStream when we create this service. Basically what we do is we first create a ZStream from a single value ZStream.succeed, and concat that one with the ZStream.fromSchedule(Schedule.spaced(tempConfig.interval.toJava)). The second one will run once every minute, based on the Schedule that we passed in through the configuration dependency. Once we start consuming this stream, at each tick, we use mapM to call makeTemperatureCall, which uses the client dependency to make a REST call, that returns the case class, which the consumer can then process (in our case, the consumer will store it in MongoDB). We use MapM here, since the makeTemperatureCall returns a ZIO: def mapM[R1 <: R, E1 >: E, O2](f: O => ZIO[R1, E1, O2]): ZStream[R1, E1, O2] Finally you can see that we use provide(Has(clock)) on the ZStream we created. Doing that eliminates the dependency of the resulting stream on Has[Clock.Service], and results in a value that, when called by the consumer, doesn’t require anything specific in the environment. We could have left out the provide but then the signature of this function would change: // with provide val temperatureStream: ZStream[Any, Throwable, Temperature] // without provide val temperatureStream: ZStream[Has[Clock.Service], Throwable, Temperature] It’s debatable whether we should leave the dependency in the signature or remove it. For me I think it is cleaner to pass the dependency during the creation of the Live layer, since it is more implementation focussed, and doesn’t really deal with any business logic needed in the environment (e.g I would add a UserContext explicitly to the trait). For completeness sake I’ll also show the implementation of the Has[Client] (the HTTP4S client): object http4sClient { object Http4sClientLive { val layer: ZLayer[Any, Throwable, Has[Client[Task]]] = { implicit val runtime: Runtime[ZEnv] = Runtime.default val res = BlazeClientBuilder[Task](runtime.platform.executor.asEC).resource.toManagedZIO ZLayer.fromManaged(res) } } } Here it isn’t really useful to create a trait and use construction based injection of dependencies, so we just create the implementation directly when we access the layer. Storing data in the database Before we connect all the different parts lets look at the mongoDB stuff. To connect to Mongo we’re going to need a MongoDBClient: object mongo { object MongoDBConnectionLive { /** Slightly different approach for when we're using case classes, since we don't really expose * the functions on the service, but want to expose a mongoDB connection directly. */ val managedMongoClient: ZManaged[Has[Configuration], Throwable, MongoClient] = for { config <- Configuration.config.toManaged_ mongoClient <- ZManaged.make(acquireConnection(config.dbConfig.endpoint))(releaseConnection(_)) } yield (mongoClient) /** Try and connect the database */ private def acquireConnection(databaseUri: String): Task[MongoClient] = ZIO.fromTry(Try { MongoClient(databaseUri) }) /** Release the connection. If an error occurs during releasing, we just ignore it * for now. It would probably be better to check the error, whether it can be ignored * and log some stuff. But for now this should be enough * * @param mongoClient the client for which we want to release the connection * @return effect that will always succeed */ private def releaseConnection(mongoClient: MongoClient): ZIO[Any, Nothing, Unit] = ZIO.fromTry(Try(mongoClient.close)).orElse(ZIO.succeed()) /** For this component, we're not going to create a case class, since the construction of this layer * can fail, and we've got a managed resource for which we want to create a connection. */ val layer: ZLayer[Has[Configuration], Throwable, Has[MongoClient]] = ZLayer.fromManaged(managedMongoClient) } } The code above should look rather familiar by now. We create a ZLayer from a managed resource. So we specify how to acquire a connection, and what to do when we release it. For the rest nothing that exiting. Once we have the above wrapped resource, we can use it in our repository implementation. object storage { /** The trait for storing. */ trait TemperatureStorage { def insert(temperature: Temperature): ZIO[Any, Throwable, Unit] def getAll(): ZIO[Any, Throwable, List[Temperature]] } /** The implementation of temp storage, very naive for now, just to show the different parts connected to one another * * @param configuration */ case class TemperatureStorageLive(configuration: Configuration, mongoClient: MongoClient) extends TemperatureStorage { val temperatureCodecProvider = Macros.createCodecProvider[Temperature]() val codecRegistry = fromRegistries(fromProviders(temperatureCodecProvider), DEFAULT_CODEC_REGISTRY) /** Try and store the temperature string * @param temp element to store * @return ZIO containing the result */ override def insert(temperature: Temperature): ZIO[Any, Throwable, Unit] = withCollection[Unit, Temperature] { collection => collection // insert the entry .insertOne(temperature) // convert to a single result, since the result is a singleObservable .toStream() .runHead // we should do some more error handling here in a real world scenario .flatMap { case Some(res) => ZIO.succeed() case None => ZIO.fail(new IllegalArgumentException("Expected result from mongodb")) } } /** Return all the elements we currently have * * @return list of all the temperatures we've got stored */ override def getAll(): ZIO[Any, Throwable, List[Temperature]] = withCollection[List[Temperature], Temperature] { _.find() .toStream() .runCollect .map(_.toList) } /** Get the database and collection to which to store. * * @param f function to call within the context of this collection * @return result of wrapped */ private def withCollection[A, T: ClassTag]( f: MongoCollection[T] => ZIO[Any, Throwable, A] ): ZIO[Any, Throwable, A] = { val collectionZIO = ZIO.fromTry(Try { // TODO: we should get the database name at least from the configuration mongoClient.getDatabase("sampleservice").withCodecRegistry(codecRegistry).getCollection[T]("temperatures") }) collectionZIO.flatMap(f) } } object TemperatureStorageLive { val layer: URLayer[Has[Configuration] with Has[MongoClient], Has[TemperatureStorage]] = (TemperatureStorageLive(_, _)).toLayer } } Disclaimer: There are already other MongoDB clients out there that use ZIO, I just created one from scratch to show how easy it really is to interop with existing libraries and tools. The code above follows the same module pattern, where we define a trait, a case class implementing the trait, and pass in any required dependencies by using the toLayer function. Most of the code above is just making sure we can work with MongoDB, and the only interesting stuff is happening in the insert and getAll functions. override def insert(temperature: Temperature): ZIO[Any, Throwable, Unit] = withCollection[Unit, Temperature] { collection => collection // insert the entry .insertOne(temperature) // convert to a single result, since the result is a singleObservable .toStream() .runHead // we should do some more error handling here in a real world scenario .flatMap { case Some(res) => ZIO.succeed() case None => ZIO.fail(new IllegalArgumentException("Expected result from mongodb")) } } When you look at the insertOne signature: def insertOne(document: TResult): SingleObservable[InsertOneResult] = wrapped.insertOne(document) You’ll see that it returns a SingleObservable. This type is a trait defined like this: trait SingleObservable[T] extends Observable[T] { ... } trait Observable[T] extends Publisher[T] { ... } Publisher is part of the org.reactivestreams library and provides a generic interface for integration different kinds of streams. This means that we can easily convert the result from the insertOne call to a ZStream by just calling toStream() (which is a function provided by ZIO). The result is that we’ve got a ZStream which will always just return a single element. To get this element we call the runHead function, which will return a ZIO[Option[T]], which we process in the normal way. And that’s already it, without any complex integration we can easily reuse existing libraries. The getAll function is pretty much the same: override def getAll(): ZIO[Any, Throwable, List[Temperature]] = withCollection[List[Temperature], Temperature] { _.find() .toStream() .runCollect .map(_.toList) Here we convert the resulting stream from the Mongo driver to a ZStream, collect all the elements in the stream using runCollect, and finally map the Chunk[T] which we get back from the runCollect to a List[T]. I was really impressed with how easy working with ZStream is, the interoperability with existing libraries, and the interaction between the ZIO types and the ZStream types. The rest endpoint to get all the data So we’ve covered pretty much everything I wanted to explain in this article. The only thing left is the REST route to access the stored data. For this we first define some routes: object Routes { trait TemperatureRoute { val routes: HttpRoutes[Task] } object TemperatureRoute { // helper function which access the correct resource from our environment, and lifts // it in an effect. val temperatureRoutes: URIO[Has[TemperatureRoute], HttpRoutes[Task]] = ZIO.access(_.get.routes) } object TemperatureRouteLive { private val dsl = Http4sDsl[Task] import dsl._ import io.circe.generic.auto._, io.circe.syntax._ import org.http4s.dsl.Http4sDsl import org.http4s.circe._ import zio.interop.catz._ /** A simple layer which returns the routes for the temperature. */ val layer: ZLayer[Has[storage.TemperatureStorage], Nothing, Has[TemperatureRoute]] = ZLayer.fromService { storage => new TemperatureRoute { val routes = HttpRoutes .of[Task] { case GET -> Root / "temperatures" => { storage.getAll().flatMap { all => Ok(all.asJson) } } } } } } } As you can see I defined these as a separate layer, which requires access to the database to get the list of the stored data. This way we can just define the routes in isolation, and to add them we just mark them as dependency for the HTTP4S server. With all this out of the way, we can start the server and make an HTTP call to get the (very boring looking) data. combining everything To start this, we first setup the layers and then define a simple program: object Application extends zio.App { // combine the configuration layer and the standard environment into // a new layer. val defaultLayer = ConfigurationLive.layer ++ ZEnv.live // the storage layer val storageLayer = defaultLayer >+> db.mongo.MongoDBConnectionLive.layer >>> storage.TemperatureStorageLive.layer val httpsLayer = storageLayer >+> TemperatureRouteLive.layer ++ defaultLayer >+> http4sServer.Http4sServerLive.layer ++ http4sClient.Http4sClientLive.layer // combine the layers into a single layer to feed into the program val applicationLayer = httpsLayer >+> tempClient.TempClientLive.layer ++ storageLayer /** Provide the layer to the program and run it * * @param args * @return */ override def run(args: List[String]): URIO[ZEnv, ExitCode] = pp.provideLayer(applicationLayer).exitCode /** Helper type to indicate which dependencies are needed to run the program */ type ProgramDeps = Has[Server] with Has[storage.TemperatureStorage] with Has[tempClient.TempClient] with Has[Client[Task]] with zio.clock.Clock with zio.console.Console /** The program that starts polling the https endpoint and write the results to * the database. */ val pp: ZIO[ProgramDeps, Throwable, Unit] = { for { // start the stream, which polls the temperature endpoint stream <- tempClient.TempClient.temperatureStream mapped = stream.mapM(temperatureString => ZIO.serviceWith[storage.TemperatureStorage](_.insert(temperatureString)) ) _ <- mapped.runDrain } yield () } } In this program the only thing we do is, that we get the stream of temperature updates (tempClient.TempClient.temperatureStream), and for each element in the stream we define that we want to store it in database using the mapM function. We run this stream by calling runDrain, which will just keep waiting for new elements in the stream and process those. When we run this, we see output like this: [zio-default-async-1] INFO org.mongodb.driver.cluster - Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms'} [cluster-ClusterId{value='60ad3ce890de0d65166e20ab', description='null'}-localhost:27017] INFO org.mongodb.driver.connection - Opened connection [connectionId{localValue:2, serverValue:7}] to localhost:27017 [cluster-ClusterId{value='60ad3ce890de0d65166e20ab', description='null'}-localhost:27017] INFO org.mongodb.driver.cluster - Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=8, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=18662711} [cluster-rtt-ClusterId{value='60ad3ce890de0d65166e20ab', description='null'}-localhost:27017] INFO org.mongodb.driver.connection - Opened connection [connectionId{localValue:1, serverValue:8}] to localhost:27017 [zio-default-async-5] INFO org.http4s.blaze.channel.nio1.NIO1SocketServerGroup - Service bound to address /127.0.0.1:8081 [zio-default-async-5] INFO org.http4s.server.blaze.BlazeServerBuilder - _ _ _ _ _ | |_| |_| |_ _ __| | | ___ | ' \ _| _| '_ \_ _(_-< |_||_\__|\__| .__/ |_|/__/ |_| [zio-default-async-5] INFO org.http4s.server.blaze.BlazeServerBuilder - http4s v1.0.0-M4 on blaze v0.14.13 started at http://127.0.0.1:8081/ Temperature(1621966031,285.05) [InnocuousThread-2] INFO org.mongodb.driver.connection - Opened connection [connectionId{localValue:3, serverValue:9}] to localhost:27017 And when we call into the webservice the result is something like this: $ curl -v localhost:8081/temperatures * Trying 127.0.0.1:8081... * TCP_NODELAY set * Connected to localhost (127.0.0.1) port 8081 (#0) > GET /temperatures HTTP/1.1 > Host: localhost:8081 > User-Agent: curl/7.68.0 > Accept: */* > * Mark bundle as not supporting multiuse < HTTP/1.1 200 OK < Content-Type: application/json < Date: Tue, 25 May 2021 18:08:54 GMT < Content-Length: 780 < * Connection #0 to host localhost left intact [{"timestamp":1621876805,"temperature":284.12},{"timestamp":1621876918,"temperature":284.1},{"timestamp":1621876805,"temperature":284.12},{"timestamp":1621876805,"temperature":284.12},{"timestamp":1621876918,"temperature":284.1},{"timestamp":1621876918,"temperature":284.1},{"timestamp":1621877523,"temperature":283.79},{"timestamp":1621877523,"temperature":283.79},{"timestamp":1621877644,"temperature":283.76},{"timestamp":1621877644,"temperature":283.76},{"timestamp":1621877644,"temperature":283.76},{"timestamp":1621877644,"temperature":283.76},{"timestamp":1621965947,"temperature":285.05},{"timestamp":1621965947,"temperature":285.05},{"timestamp":1621966031,"temperature":285.05},{"timestamp":1621966031,"temperature":285.05},{"timestamp":1621966031,"temperature":285.05}]Exploring ZIO - Part I2021-05-16T00:00:00+02:002021-05-16T00:00:00+02:00http://www.smartjava.org/content/exploring-zio-part-1<p>I’ve done a lot of Kotlin for the last two years, and have mainly followed Scala and done some pet projects. Since a lot of interesting stuff has been going on in the Scala space for the last couple of months, I want to start writing some more about it, and my experiences with it. So expect some articles on Scala 3 and ZIO. For the first of these set of articles, I’d like to focus a bit on ZIO. I’m exploring the documentation, watched some presentations, but nothing works as best, as just writing a simple application with it.</p>
<p>So in this first part on ZIO, I’m going to look at a simple layered ZIO application that (for now) consists out of the following:</p>
<ul>
<li><strong>Loading Configuration</strong>: How to load configuration (<a href="https://github.com/pureconfig/pureconfig">PureConfig</a>), and provide that as a <code class="highlighter-rouge">ZLayer</code> to other services.</li>
<li><strong>REST endpoint with HTTP4s</strong>: I’ll add a very simple REST layer (only the basics for now), which uses information from the configuration to start a server on a specific host and port.</li>
</ul>
<p>In the end we should have a simple application that starts an HTTP Server, loads configuration in a safe way, and provides us with a simple application that we can build upon for future articles, when we explore more features of ZIO.</p>
<h2 id="zio-project-setup">ZIO Project setup</h2>
<p>If you want to play along, everything should work out of the box with <a href="https://scalameta.org/metals/">Metals</a>:</p>
<p><img src="../../images/zio_vscode.png" alt="ZIO in VSCode" /></p>
<p>Before we can do anything, we need to define the dependencies (<code class="highlighter-rouge">build.sbt</code>). When I started playing around with this, there were some dependencies issues between the version of cats, the version of HTTP4S and the <code class="highlighter-rouge">zio-cats-interop</code> stuff, so after some checking, the following is a good starting point (I’ll have a look and update this to a newer version in the future).</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">import</span> <span class="nn">Dependencies._</span>
<span class="nc">ThisBuild</span> <span class="o">/</span> <span class="n">scalaVersion</span> <span class="o">:=</span> <span class="s">"2.13.4"</span>
<span class="nc">ThisBuild</span> <span class="o">/</span> <span class="n">version</span> <span class="o">:=</span> <span class="s">"0.1.0-SNAPSHOT"</span>
<span class="nc">ThisBuild</span> <span class="o">/</span> <span class="n">organization</span> <span class="o">:=</span> <span class="s">"com.example"</span>
<span class="nc">ThisBuild</span> <span class="o">/</span> <span class="n">organizationName</span> <span class="o">:=</span> <span class="s">"example"</span>
<span class="c1">// need to use an older version, since the newest version of http4s
// supports the latest cats version, while the zio-interop-cats one
// doesn't support this yet.
</span><span class="k">val</span> <span class="nc">Http4sVersion</span> <span class="k">=</span> <span class="s">"1.0.0-M4"</span>
<span class="k">lazy</span> <span class="k">val</span> <span class="n">root</span> <span class="k">=</span> <span class="o">(</span><span class="n">project</span> <span class="n">in</span> <span class="n">file</span><span class="o">(</span><span class="s">"."</span><span class="o">))</span>
<span class="o">.</span><span class="n">settings</span><span class="o">(</span>
<span class="n">name</span> <span class="o">:=</span> <span class="s">"zio-playground"</span><span class="o">,</span>
<span class="n">libraryDependencies</span> <span class="o">+=</span> <span class="n">scalaTest</span> <span class="o">%</span> <span class="nc">Test</span><span class="o">,</span>
<span class="n">libraryDependencies</span> <span class="o">++=</span> <span class="nc">Seq</span><span class="o">(</span>
<span class="s">"org.http4s"</span> <span class="o">%%</span> <span class="s">"http4s-blaze-server"</span> <span class="o">%</span> <span class="nc">Http4sVersion</span><span class="o">,</span>
<span class="s">"org.http4s"</span> <span class="o">%%</span> <span class="s">"http4s-dsl"</span> <span class="o">%</span> <span class="nc">Http4sVersion</span><span class="o">,</span>
<span class="s">"dev.zio"</span> <span class="o">%%</span> <span class="s">"zio"</span> <span class="o">%</span> <span class="s">"1.0.6"</span><span class="o">,</span>
<span class="s">"dev.zio"</span> <span class="o">%%</span> <span class="s">"zio-streams"</span> <span class="o">%</span> <span class="s">"1.0.6"</span><span class="o">,</span>
<span class="s">"dev.zio"</span> <span class="o">%%</span> <span class="s">"zio-interop-cats"</span> <span class="o">%</span> <span class="s">"2.4.0.0"</span><span class="o">,</span>
<span class="s">"com.github.pureconfig"</span> <span class="o">%%</span> <span class="s">"pureconfig"</span> <span class="o">%</span> <span class="s">"0.15.0"</span><span class="o">,</span>
<span class="s">"org.slf4j"</span> <span class="o">%</span> <span class="s">"slf4j-api"</span> <span class="o">%</span> <span class="s">"1.7.5"</span><span class="o">,</span>
<span class="s">"org.slf4j"</span> <span class="o">%</span> <span class="s">"slf4j-simple"</span> <span class="o">%</span> <span class="s">"1.7.5"</span>
<span class="s">"dev.zio"</span> <span class="o">%%</span> <span class="s">"zio-test"</span> <span class="o">%</span> <span class="s">"1.0.6"</span> <span class="o">%</span> <span class="s">"test"</span><span class="o">,</span>
<span class="s">"dev.zio"</span> <span class="o">%%</span> <span class="s">"zio-test-sbt"</span> <span class="o">%</span> <span class="s">"1.0.6"</span> <span class="o">%</span> <span class="s">"test"</span>
<span class="o">)</span>
<span class="o">)</span>
</code></pre></div></div>
<p>Looking at the dependencies above, nothing out of the ordinary. Note the last two dependencies which we’ll use for testing part of the application.</p>
<h2 id="providing-a-configuration">Providing a configuration</h2>
<p>For the configuration part we’re going to use (<a href="https://github.com/pureconfig/pureconfig">PureConfig</a>), which is a typesafe, secure way of loading configurations. How this library works is a bit out of scope, but basically what you do is, you define a set of case classes defining the configuration model, and fill them by loading a configuration file. For this part we have a very basic configuration file:</p>
<div class="language-conf highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">api</span>-<span class="n">config</span> {
<span class="n">endpoint</span> = <span class="s2">"localhost"</span>
<span class="n">port</span> = <span class="m">8081</span>
}
</code></pre></div></div>
<p>To load this configuration we call <code class="highlighter-rouge">ConfigSource.default.load[Config]</code>, this returns us with an <code class="highlighter-rouge">Either</code> where we have a list of configuration exceptions, or the loaded configuration. To wrap this in ZIO we need to lift it into a <code class="highlighter-rouge">ZIO</code> type. We’ll use the <code class="highlighter-rouge">Task</code> type from ZIO for this. A <code class="highlighter-rouge">Task</code> is one of the default type aliases:</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">type</span> <span class="kt">IO</span><span class="o">[</span><span class="kt">+E</span>, <span class="kt">+A</span><span class="o">]</span> <span class="k">=</span> <span class="nc">ZIO</span><span class="o">[</span><span class="kt">Any</span>, <span class="kt">E</span>, <span class="kt">A</span><span class="o">]</span> <span class="c1">// Succeed with an `A`, may fail with `E` , no requirements.
</span> <span class="k">type</span> <span class="kt">Task</span><span class="o">[</span><span class="kt">+A</span><span class="o">]</span> <span class="k">=</span> <span class="nc">ZIO</span><span class="o">[</span><span class="kt">Any</span>, <span class="kt">Throwable</span>, <span class="kt">A</span><span class="o">]</span> <span class="c1">// Succeed with an `A`, may fail with `Throwable`, no requirements.
</span> <span class="k">type</span> <span class="kt">RIO</span><span class="o">[</span><span class="kt">-R</span>, <span class="kt">+A</span><span class="o">]</span> <span class="k">=</span> <span class="nc">ZIO</span><span class="o">[</span><span class="kt">R</span>, <span class="kt">Throwable</span>, <span class="kt">A</span><span class="o">]</span> <span class="c1">// Succeed with an `A`, may fail with `Throwable`, requires an `R`.
</span> <span class="k">type</span> <span class="kt">UIO</span><span class="o">[</span><span class="kt">+A</span><span class="o">]</span> <span class="k">=</span> <span class="nc">ZIO</span><span class="o">[</span><span class="kt">Any</span>, <span class="kt">Nothing</span>, <span class="kt">A</span><span class="o">]</span> <span class="c1">// Succeed with an `A`, cannot fail , no requirements.
</span> <span class="k">type</span> <span class="kt">URIO</span><span class="o">[</span><span class="kt">-R</span>, <span class="kt">+A</span><span class="o">]</span> <span class="k">=</span> <span class="nc">ZIO</span><span class="o">[</span><span class="kt">R</span>, <span class="kt">Nothing</span>, <span class="kt">A</span><span class="o">]</span> <span class="c1">// Succeed with an `A`, cannot fail , requires an `R`.
</span></code></pre></div></div>
<p>So we’ve got an operation that can either fail with a <code class="highlighter-rouge">Throwable</code> or succeed with an <code class="highlighter-rouge">A</code>. To lift our <code class="highlighter-rouge">Either</code> we just call <code class="highlighter-rouge">fromEither</code>:</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nc">Task</span><span class="o">.</span><span class="n">fromEither</span><span class="o">(</span>
<span class="nc">ConfigSource</span><span class="o">.</span><span class="n">default</span>
<span class="o">.</span><span class="n">load</span><span class="o">[</span><span class="kt">Config</span><span class="o">]</span>
<span class="o">.</span><span class="n">left</span>
<span class="o">.</span><span class="n">map</span><span class="o">(</span><span class="n">pureconfig</span><span class="o">.</span><span class="n">error</span><span class="o">.</span><span class="nc">ConfigReaderException</span><span class="o">.</span><span class="n">apply</span><span class="o">)</span>
<span class="o">)</span>
</code></pre></div></div>
<p>Since the <code class="highlighter-rouge">load</code> returns a list of <code class="highlighter-rouge">ConfigReaderFailures</code>, we just map the left part into an exception, before we can convert it into a ZIO <code class="highlighter-rouge">Task</code>. Now that we’ve got this configuration, we can use it in for-comprehensions, pass it to other components that need it and anything else you want to do with it. For this example we’re going one step further, and we’ll define the loading of the configuration as a service, which we can define as a dependency for other services.
For ZIO the standard way of doing this is by using <a href="https://zio.dev/docs/datatypes/core/zlayer">ZLayer</a>. I found the documentation a bit hard to follow, but once you start playing around with it, it becomes a lot easier to understand. Below I’ve put the complete source for the <code class="highlighter-rouge">configuration</code> service, with inline comments explaining the various steps. Note that this structure is the proposed convention from ZIO, so it’s something you’ll see coming back in multiple sources.</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">import</span> <span class="nn">zio.Task</span>
<span class="k">import</span> <span class="nn">zio.Has</span>
<span class="k">import</span> <span class="nn">zio.ZIO</span>
<span class="k">import</span> <span class="nn">pureconfig.ConfigSource</span>
<span class="k">import</span> <span class="nn">pureconfig.generic.auto._</span>
<span class="k">import</span> <span class="nn">zio.ZLayer</span>
<span class="k">import</span> <span class="nn">zio.Layer</span>
<span class="c1">// define the domain model used for configuration
</span><span class="k">case</span> <span class="k">class</span> <span class="nc">Config</span><span class="o">(</span><span class="n">apiConfig</span><span class="k">:</span> <span class="kt">ApiConfig</span><span class="o">)</span>
<span class="k">case</span> <span class="k">class</span> <span class="nc">ApiConfig</span><span class="o">(</span><span class="n">endpoint</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">port</span><span class="k">:</span> <span class="kt">Int</span><span class="o">)</span>
<span class="c1">// define the configuration dependency to inject into the environment
// and the related functions
</span><span class="k">object</span> <span class="nc">configuration</span> <span class="o">{</span>
<span class="c1">// Custom type to inject
</span> <span class="k">type</span> <span class="kt">Configuration</span> <span class="o">=</span> <span class="n">zio</span><span class="o">.</span><span class="nc">Has</span><span class="o">[</span><span class="kt">Configuration.Service</span><span class="o">]</span>
<span class="c1">// contains the service definition, and multiple implementations
</span> <span class="c1">// based on the environment we want to run in.
</span> <span class="k">object</span> <span class="nc">Configuration</span> <span class="o">{</span>
<span class="c1">// we just have the load function inside the service to load
</span> <span class="c1">// some configuration
</span> <span class="k">trait</span> <span class="nc">Service</span> <span class="o">{</span>
<span class="k">val</span> <span class="n">load</span><span class="k">:</span> <span class="kt">Task</span><span class="o">[</span><span class="kt">Config</span><span class="o">]</span>
<span class="o">}</span>
<span class="c1">// we can have multiple implementations of this service. This is the
</span> <span class="c1">// live implementation, which loads the config through pureconfig
</span> <span class="k">val</span> <span class="n">live</span><span class="k">:</span> <span class="kt">Layer</span><span class="o">[</span><span class="kt">Nothing</span>, <span class="kt">Configuration</span><span class="o">]</span> <span class="k">=</span> <span class="nc">ZLayer</span><span class="o">.</span><span class="n">succeed</span> <span class="o">{</span>
<span class="k">new</span> <span class="nc">Service</span> <span class="o">{</span>
<span class="k">override</span> <span class="k">val</span> <span class="n">load</span><span class="k">:</span> <span class="kt">Task</span><span class="o">[</span><span class="kt">Config</span><span class="o">]</span> <span class="k">=</span> <span class="nc">Task</span><span class="o">.</span><span class="n">fromEither</span><span class="o">(</span>
<span class="nc">ConfigSource</span><span class="o">.</span><span class="n">default</span>
<span class="o">.</span><span class="n">load</span><span class="o">[</span><span class="kt">Config</span><span class="o">]</span>
<span class="o">.</span><span class="n">left</span>
<span class="o">.</span><span class="n">map</span><span class="o">(</span><span class="n">pureconfig</span><span class="o">.</span><span class="n">error</span><span class="o">.</span><span class="nc">ConfigReaderException</span><span class="o">.</span><span class="n">apply</span><span class="o">)</span>
<span class="o">)</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="c1">// helper function which access the correct resource from our environment.
</span> <span class="k">val</span> <span class="n">load</span><span class="k">:</span> <span class="kt">ZIO</span><span class="o">[</span><span class="kt">Configuration</span>, <span class="kt">Throwable</span>, <span class="kt">Config</span><span class="o">]</span> <span class="k">=</span> <span class="nc">ZIO</span><span class="o">.</span><span class="n">accessM</span><span class="o">(</span><span class="k">_</span><span class="o">.</span><span class="n">get</span><span class="o">.</span><span class="n">load</span><span class="o">)</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>What we do here is first define the case classes that map to the configuration file, that is just something specific to pure-config. Next we define the <code class="highlighter-rouge">configuration</code> object, where we define the ZIO / ZLayer specific stuff. We define a <code class="highlighter-rouge">Configuration</code> type, which is the type we can have other services depend on. This is the type which uniquely identifies that functionality offered by this service. To define this we use <code class="highlighter-rouge">zio.Has[A]</code>, which following the scaladoc, does the following:</p>
<blockquote>
<p>The trait <code class="highlighter-rouge">Has[A]</code> is used with ZIO environment to express an effect’s
dependency on a service of type <code class="highlighter-rouge">A</code>. For example,
<code class="highlighter-rouge">RIO[Has[Console.Service], Unit]</code> is an effect that requires a
<code class="highlighter-rouge">Console.Service</code> service. Inside the ZIO library, type aliases are provided
as shorthands for common services, e.g.:</p>
</blockquote>
<p>After the type definition, we define a trait with the specific functions provided by this service. In our case this is just the load function (we’d actually could do better by loading during service creation, and assign it to a lazy val or use <code class="highlighter-rouge">memomize</code>). The next step we need to do is define the different implementation we have of our service. We could have specific ones for testing, for specific environments and so on. In this case we have one, for which it is the convention to call that one <code class="highlighter-rouge">live</code>:</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">val</span> <span class="n">live</span><span class="k">:</span> <span class="kt">Layer</span><span class="o">[</span><span class="kt">Nothing</span>, <span class="kt">Configuration</span><span class="o">]</span> <span class="k">=</span> <span class="nc">ZLayer</span><span class="o">.</span><span class="n">succeed</span> <span class="o">{</span>
<span class="k">new</span> <span class="nc">Service</span> <span class="o">{</span>
<span class="k">override</span> <span class="k">val</span> <span class="n">load</span><span class="k">:</span> <span class="kt">Task</span><span class="o">[</span><span class="kt">Config</span><span class="o">]</span> <span class="k">=</span> <span class="nc">Task</span><span class="o">.</span><span class="n">fromEither</span><span class="o">(</span>
<span class="nc">ConfigSource</span><span class="o">.</span><span class="n">default</span>
<span class="o">.</span><span class="n">load</span><span class="o">[</span><span class="kt">Config</span><span class="o">]</span>
<span class="o">.</span><span class="n">left</span>
<span class="o">.</span><span class="n">map</span><span class="o">(</span><span class="n">pureconfig</span><span class="o">.</span><span class="n">error</span><span class="o">.</span><span class="nc">ConfigReaderException</span><span class="o">.</span><span class="n">apply</span><span class="o">)</span>
<span class="o">)</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Here we load the configuration, and lift it into the <code class="highlighter-rouge">Task</code>. We create an anonymous instance, and return that as the result of the <code class="highlighter-rouge">ZLayer.succeed</code> function. <code class="highlighter-rouge">ZLayer.succeed</code> means that service creation of this instance will always succeed. In our case that is true, since we only load the config at the point it is needed. There are other <code class="highlighter-rouge">ZLayer</code> functions which allow you to create managed resources, or where the creation of the service returns an effect (e.g a <code class="highlighter-rouge">Task</code>). For now though, since nothing can go wrong here, we use <code class="highlighter-rouge">ZLayer.succeed</code>.</p>
<p>The final part of this service, is that we provide a convenience method called <code class="highlighter-rouge">load</code>, which mirrors the <code class="highlighter-rouge">load</code> function from our service:</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">val</span> <span class="n">load</span><span class="k">:</span> <span class="kt">ZIO</span><span class="o">[</span><span class="kt">Configuration</span>, <span class="kt">Throwable</span>, <span class="kt">Config</span><span class="o">]</span> <span class="k">=</span> <span class="nc">ZIO</span><span class="o">.</span><span class="n">accessM</span><span class="o">(</span><span class="k">_</span><span class="o">.</span><span class="n">get</span><span class="o">.</span><span class="n">load</span><span class="o">)</span>
</code></pre></div></div>
<p>This means that if there is a <code class="highlighter-rouge">Configuration</code> in the provided environment, we can use this helper function to directly access the load function, and load the configuration. We’ll show you how this is used further down in this article, when we start tying services together.</p>
<h2 id="testing-a-configuration">Testing a configuration</h2>
<p>So far we’ve seen quite some code, but how do we now that this will do what it needs to do. For this ZIO provides some testing libraries, where we can test whether the service we just created does what it needs to do. So let’s write a test:</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">import</span> <span class="nn">org.scalatest.flatspec.AnyFlatSpec</span>
<span class="k">import</span> <span class="nn">org.scalatest.matchers.should.Matchers</span>
<span class="k">import</span> <span class="nn">zio.test.DefaultRunnableSpec</span>
<span class="k">import</span> <span class="nn">zio.test._</span>
<span class="k">import</span> <span class="nn">Assertion._</span>
<span class="k">object</span> <span class="nc">LiveConfigLoaderSpec</span> <span class="k">extends</span> <span class="nc">DefaultRunnableSpec</span> <span class="o">{</span>
<span class="k">override</span> <span class="k">def</span> <span class="n">spec</span><span class="k">:</span> <span class="kt">ZSpec</span><span class="o">[</span><span class="kt">Environment</span>, <span class="kt">Failure</span><span class="o">]</span> <span class="k">=</span>
<span class="n">suite</span><span class="o">(</span><span class="s">"ConfigLoaderSpec"</span><span class="o">)(</span>
<span class="n">testM</span><span class="o">(</span><span class="s">"Load the configuration"</span><span class="o">)</span> <span class="o">{</span>
<span class="k">for</span> <span class="o">{</span>
<span class="n">config</span> <span class="k"><-</span> <span class="n">configuration</span><span class="o">.</span><span class="nc">Configuration</span><span class="o">.</span><span class="n">load</span>
<span class="o">}</span> <span class="k">yield</span> <span class="o">{</span>
<span class="n">assert</span><span class="o">(</span><span class="n">config</span><span class="o">.</span><span class="n">apiConfig</span><span class="o">.</span><span class="n">port</span><span class="o">)(</span><span class="n">equalTo</span><span class="o">(</span><span class="mi">8081</span><span class="o">))</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">).</span><span class="n">provideCustomLayer</span><span class="o">(</span><span class="n">configuration</span><span class="o">.</span><span class="nc">Configuration</span><span class="o">.</span><span class="n">live</span><span class="o">)</span>
<span class="o">}</span>
</code></pre></div></div>
<p>What we do here is that we call the <code class="highlighter-rouge">load</code> function we just defined in the companion object, and based on the passed in layer, we can check whether it works. In this example we pass in the <code class="highlighter-rouge">configuration.Configuration.live</code> layer. So now when we call <code class="highlighter-rouge">configuration.Configuration.load</code>, we should call the load of the pureconfig based configuration.</p>
<p>Running the above test will show something like this:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>+ ConfigLoaderSpec
+ Load the configuration
Ran 1 test in 751 ms: 1 succeeded, 0 ignored, 0 failed
</code></pre></div></div>
<p>At this point we’ve got a simple configuration service, but we haven’t got an application yet. So before we move on the http4s stuff, we’re going to set up a minimal application.</p>
<h2 id="minimal-application">Minimal application</h2>
<p>Once again, I’ll show the annotated code, and then go into some of the details:</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">import</span> <span class="nn">zio.ExitCode</span>
<span class="k">import</span> <span class="nn">zio.URIO</span>
<span class="k">import</span> <span class="nn">zio.ZEnv</span>
<span class="k">import</span> <span class="nn">configuration.Configuration</span>
<span class="k">import</span> <span class="nn">example.services.http4sServer</span>
<span class="k">import</span> <span class="nn">zio.ZIO</span>
<span class="cm">/** Base entry point for a ZIO app, which runs the logic within
* a specified environment
*/</span>
<span class="k">object</span> <span class="nc">Application</span> <span class="k">extends</span> <span class="n">zio</span><span class="o">.</span><span class="nc">App</span> <span class="o">{</span>
<span class="c1">// now run the application, by providing it with the application layer
</span> <span class="k">override</span> <span class="k">def</span> <span class="n">run</span><span class="o">(</span><span class="n">args</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">String</span><span class="o">])</span><span class="k">:</span> <span class="kt">URIO</span><span class="o">[</span><span class="kt">ZEnv</span>, <span class="kt">ExitCode</span><span class="o">]</span> <span class="k">=</span>
<span class="n">myAppLogic</span><span class="o">.</span><span class="n">provideLayer</span><span class="o">(</span><span class="nc">Configuration</span><span class="o">.</span><span class="n">live</span><span class="o">).</span><span class="n">exitCode</span>
<span class="c1">// we can do more stuff here. For now just load the config, cause why not.
</span> <span class="k">val</span> <span class="n">myAppLogic</span> <span class="k">=</span>
<span class="k">for</span> <span class="o">{</span>
<span class="n">config</span> <span class="k"><-</span> <span class="nc">Configuration</span><span class="o">.</span><span class="n">load</span>
<span class="o">}</span> <span class="k">yield</span> <span class="o">(</span>
<span class="n">println</span><span class="o">(</span><span class="n">config</span><span class="o">)</span>
<span class="o">)</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Here we extend from <code class="highlighter-rouge">zio.App</code> which gives us the <code class="highlighter-rouge">override def run(args: List[String]): URIO[ZEnv, ExitCode]</code> function to implement as entrypoint in the application. What we do in this entry point is call our program, and when we do that we provide a set of layers containing our services. In this example we just provide the layer we just defined, where we can load the configuration.
In out program <code class="highlighter-rouge">myAppLogic</code> we load the configuration (which is present in the environment of our ZIO type.). The result is very simply that we see this:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ sbt run
[info] welcome to sbt 1.5.0 (AdoptOpenJDK Java 11.0.7)
[info] loading settings for project zio-playground-build-build from metals.sbt ...
[info] loading project definition from /home/jos/dev/git/smartjava/zio-playground/project/project
[info] loading settings for project zio-playground-build from metals.sbt ...
[info] loading project definition from /home/jos/dev/git/smartjava/zio-playground/project
[success] Generated .bloop/zio-playground-build.json
[success] Total time: 1 s, completed May 17, 2021, 4:56:17 PM
[info] loading settings for project root from build.sbt ...
[info] set current project to zio-playground (in build file:/home/jos/dev/git/smartjava/zio-playground/)
[info] running example.Application
Config(ApiConfig(localhost,8081))
</code></pre></div></div>
<p>What we did above is pretty much the same as we did in the test. We provide a program to execute, and when we execute that program we pass in a set of services that can be accessed through the environment.</p>
<h2 id="adding-the-http4s-server-dependency">Adding the HTTP4S server dependency</h2>
<p>For the last part of this article, we’ll also add a very simple HTTP4S server that will just return ok (we’ll look at the mapping to case classes later). For the HTTP4S server we want it to use the configuration layer we defined earlier. To get the HTTP4S server running we need to take a couple of extra steps to make it work correctly with ZIO.</p>
<p>To create the HTTPServer we use the following function on the <code class="highlighter-rouge">BlazeServerBuilder</code></p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="n">apply</span><span class="o">[</span><span class="kt">F</span><span class="o">[</span><span class="k">_</span><span class="o">]](</span><span class="n">executionContext</span><span class="k">:</span> <span class="kt">ExecutionContext</span><span class="o">)(</span><span class="k">implicit</span>
<span class="n">F</span><span class="k">:</span> <span class="kt">ConcurrentEffect</span><span class="o">[</span><span class="kt">F</span><span class="o">],</span>
<span class="n">timer</span><span class="k">:</span> <span class="kt">Timer</span><span class="o">[</span><span class="kt">F</span><span class="o">])</span><span class="k">:</span> <span class="kt">BlazeServerBuilder</span><span class="o">[</span><span class="kt">F</span><span class="o">]</span>
</code></pre></div></div>
<p>If you look at this signature you can see that, besides an explicit <code class="highlighter-rouge">executionContext</code> we also need an implicit <code class="highlighter-rouge">Timer[F]</code> and implicit <code class="highlighter-rouge">ConcurrentEffect[F]</code>, where <code class="highlighter-rouge">F</code> is the type we want to work with. Since we’re already working with a <code class="highlighter-rouge">Task</code> in the config service, lets do that as well for this part. So we want to use the builder like this: <code class="highlighter-rouge">BlazeServerBuilder[Task](someExecutionContext)</code>.</p>
<p>This won’t work however, since we don’t have the correct implicits in scope, and don’t use the types from Cats. Luckily zio provides an interop library that provides us with the correct implicits. For instance the <code class="highlighter-rouge">ConcurrentEffect[Task]</code> and <code class="highlighter-rouge">Timer[Task</code> are provided by this implicit function from the <code class="highlighter-rouge">CatsEffectInstances</code> class:</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">implicit</span> <span class="k">final</span> <span class="k">def</span> <span class="n">taskEffectInstance</span><span class="o">[</span><span class="kt">R</span><span class="o">](</span><span class="k">implicit</span> <span class="n">runtime</span><span class="k">:</span> <span class="kt">Runtime</span><span class="o">[</span><span class="kt">R</span><span class="o">])</span><span class="k">:</span> <span class="kt">effect.ConcurrentEffect</span><span class="o">[</span><span class="kt">RIO</span><span class="o">[</span><span class="kt">R</span>, <span class="kt">*</span><span class="o">]]</span> <span class="k">=</span>
<span class="k">new</span> <span class="nc">CatsConcurrentEffect</span><span class="o">[</span><span class="kt">R</span><span class="o">](</span><span class="n">runtime</span><span class="o">)</span>
<span class="k">implicit</span> <span class="k">final</span> <span class="k">def</span> <span class="n">zioTimer</span><span class="o">[</span><span class="kt">R</span> <span class="k"><:</span> <span class="kt">Clock</span>, <span class="kt">E</span><span class="o">]</span><span class="k">:</span> <span class="kt">effect.Timer</span><span class="o">[</span><span class="kt">ZIO</span><span class="o">[</span><span class="kt">R</span>, <span class="kt">E</span>, <span class="kt">*</span><span class="o">]]</span> <span class="k">=</span>
<span class="n">zioTimer0</span><span class="o">.</span><span class="n">asInstanceOf</span><span class="o">[</span><span class="kt">effect.Timer</span><span class="o">[</span><span class="kt">ZIO</span><span class="o">[</span><span class="kt">R</span>, <span class="kt">E</span>, <span class="kt">*</span><span class="o">]]]</span>
</code></pre></div></div>
<p>As you can see the <code class="highlighter-rouge">taskEffectInstance</code> needs a <code class="highlighter-rouge">Runtime</code> as implicit parameter, so we need to make sure that our service that we’re defining has access to the ZIO Runtime. And looking back at the initial <code class="highlighter-rouge">apply</code> function for the <code class="highlighter-rouge">BlazeServerBuilder</code> we can also see that it needs an <code class="highlighter-rouge">executionContext</code>. Luckily an <code class="highlighter-rouge">executionContext</code> is also provided by the ZIO Runtime.</p>
<p>Now that we now which dependencies we need we can create the service. First we’ll look at the basic structure, and then a bit closer at how to create the HTTP4s server:</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">import</span> <span class="nn">org.http4s.server.Server</span>
<span class="k">import</span> <span class="nn">zio._</span>
<span class="k">import</span> <span class="nn">example.configuration.Configuration</span>
<span class="k">object</span> <span class="nc">http4sServer</span> <span class="o">{</span>
<span class="k">type</span> <span class="kt">Http4sServer</span> <span class="o">=</span> <span class="n">zio</span><span class="o">.</span><span class="nc">Has</span><span class="o">[</span><span class="kt">Http4sServer.Service</span><span class="o">]</span>
<span class="k">object</span> <span class="nc">Http4sServer</span> <span class="o">{</span>
<span class="k">type</span> <span class="kt">Service</span> <span class="o">=</span> <span class="nc">Has</span><span class="o">[</span><span class="kt">Server</span><span class="o">]</span>
<span class="k">val</span> <span class="n">live</span><span class="k">:</span> <span class="kt">ZLayer</span><span class="o">[</span><span class="kt">ZEnv</span> <span class="kt">with</span> <span class="kt">Configuration</span>, <span class="kt">Throwable</span>, <span class="kt">Service</span><span class="o">]</span> <span class="k">=</span> <span class="o">...</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>The code above defines our server, and the signature for the <code class="highlighter-rouge">live</code> instance. As we mentioned we want to use information from the <code class="highlighter-rouge">Configuration</code> dependency to configure the service, and we need to have access to the zio runtime to correctly set up HTTP4s. ZIO has helper functions for creating the runtime, so lets look at the implementation of the <code class="highlighter-rouge">live</code> instance.</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">val</span> <span class="n">live</span><span class="k">:</span> <span class="kt">ZLayer</span><span class="o">[</span><span class="kt">ZEnv</span> <span class="kt">with</span> <span class="kt">Configuration</span>, <span class="kt">Throwable</span>, <span class="kt">Service</span><span class="o">]</span> <span class="k">=</span>
<span class="nc">ZLayer</span><span class="o">.</span><span class="n">fromManaged</span> <span class="o">{</span>
<span class="k">for</span> <span class="o">{</span>
<span class="c1">// we should probably cache this, and make this a function, where we
</span> <span class="c1">// pass in the relevant configuration
</span> <span class="n">config</span> <span class="k"><-</span> <span class="nc">Configuration</span><span class="o">.</span><span class="n">load</span><span class="o">.</span><span class="n">toManaged_</span>
<span class="c1">// The implicit runtime provided the implicits needed to create the ConcurrentEffect.
</span> <span class="c1">// This is done by the zio / cats interop imports.
</span> <span class="n">server</span> <span class="k"><-</span> <span class="nc">ZManaged</span><span class="o">.</span><span class="n">runtime</span><span class="o">[</span><span class="kt">ZEnv</span><span class="o">].</span><span class="n">flatMap</span> <span class="o">{</span>
<span class="k">implicit</span> <span class="n">runtime</span><span class="k">:</span> <span class="kt">Runtime</span><span class="o">[</span><span class="kt">ZEnv</span><span class="o">]</span> <span class="k">=></span>
<span class="nc">BlazeServerBuilder</span><span class="o">[</span><span class="kt">Task</span><span class="o">](</span><span class="n">runtime</span><span class="o">.</span><span class="n">platform</span><span class="o">.</span><span class="n">executor</span><span class="o">.</span><span class="n">asEC</span><span class="o">)</span>
<span class="o">.</span><span class="n">bindHttp</span><span class="o">(</span><span class="n">config</span><span class="o">.</span><span class="n">apiConfig</span><span class="o">.</span><span class="n">port</span><span class="o">,</span> <span class="n">config</span><span class="o">.</span><span class="n">apiConfig</span><span class="o">.</span><span class="n">endpoint</span><span class="o">)</span>
<span class="o">.</span><span class="n">withHttpApp</span><span class="o">(</span><span class="nc">Routes</span><span class="o">.</span><span class="n">routes</span><span class="o">)</span>
<span class="o">.</span><span class="n">resource</span>
<span class="o">.</span><span class="n">toManagedZIO</span>
<span class="o">}</span>
<span class="o">}</span> <span class="k">yield</span> <span class="o">(</span><span class="n">server</span><span class="o">)</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Before we look at the <code class="highlighter-rouge">fromManaged</code>, let’s first look at the way we create the <code class="highlighter-rouge">server</code>. As we mentioned we need access the runtime, ZIO provides a useful helper for this called <code class="highlighter-rouge">ZManaged.runtime</code>, where we create a runtime based on the passed in type. Using this runtime, we can create the <code class="highlighter-rouge">BlazeServerBuilder</code>. When using cats we would use the resource to safely start and stop the server. In ZIO world this is called a <code class="highlighter-rouge">ZManaged</code>:</p>
<blockquote>
<p>A <code class="highlighter-rouge">ZManaged[R, E, A]</code> is a managed resource of type <code class="highlighter-rouge">A</code>, which may be used by
invoking the <code class="highlighter-rouge">use</code> method of the resource. The resource will be automatically
acquired before the resource is used, and automatically released after the
resource is used.</p>
</blockquote>
<p>The resource from the call to <code class="highlighter-rouge">.resource</code> can be converted to a <code class="highlighter-rouge">ZManaged</code> by calling <code class="highlighter-rouge">toManagedZIO</code>. Since this returns a <code class="highlighter-rouge">ZManaged</code> type, we also need to make sure that the other types in our for-comprehension are also <code class="highlighter-rouge">ZManaged</code>s. So we use the <code class="highlighter-rouge">toManaged_</code> function to convert the result from the <code class="highlighter-rouge">Configuration.load</code> to a <code class="highlighter-rouge">ZManaged</code>. At the end of the for-comprehension we can then convert the resulting <code class="highlighter-rouge">ZManaged</code> into a <code class="highlighter-rouge">ZLayer</code> by calling the <code class="highlighter-rouge">fromManaged</code>.</p>
<p>This seems like a lot of work, but it nicely shows how well ZIO can play with existing Cats types.</p>
<h2 id="defining-a-minimal-api-and-running-the-application">Defining a minimal API and running the application</h2>
<p>Without going too much into detail on the http4s specifics, the following are the routes we support for now (don’t look at the implementation, since it’s just some placeholders for future implementation)</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">object</span> <span class="nc">Routes</span> <span class="o">{</span>
<span class="k">private</span> <span class="k">val</span> <span class="n">dsl</span> <span class="k">=</span> <span class="nc">Http4sDsl</span><span class="o">[</span><span class="kt">Task</span><span class="o">]</span>
<span class="k">import</span> <span class="nn">dsl._</span>
<span class="k">def</span> <span class="n">routes</span> <span class="k">=</span> <span class="nc">HttpRoutes</span>
<span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">Task</span><span class="o">]</span> <span class="o">{</span>
<span class="k">case</span> <span class="nc">GET</span> <span class="o">-></span> <span class="nc">Root</span> <span class="o">/</span> <span class="s">"users"</span> <span class="o">/</span> <span class="nc">IntVar</span><span class="o">(</span><span class="n">id</span><span class="o">)</span> <span class="k">=></span> <span class="o">{</span>
<span class="nc">Created</span><span class="o">(</span><span class="s">"A value here"</span><span class="o">)</span>
<span class="o">}</span>
<span class="k">case</span> <span class="nc">POST</span> <span class="o">-></span> <span class="nc">Root</span> <span class="o">/</span> <span class="s">"users"</span> <span class="k">=></span> <span class="o">{</span>
<span class="nc">Created</span><span class="o">(</span><span class="s">"And another one here"</span><span class="o">)</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">.</span><span class="n">orNotFound</span>
<span class="o">}</span>
</code></pre></div></div>
<p>And with some small updates our main program looks like this:</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">package</span> <span class="nn">example</span>
<span class="k">import</span> <span class="nn">zio.ExitCode</span>
<span class="k">import</span> <span class="nn">zio.URIO</span>
<span class="k">import</span> <span class="nn">zio.ZEnv</span>
<span class="k">import</span> <span class="nn">configuration.Configuration</span>
<span class="k">import</span> <span class="nn">example.services.http4sServer</span>
<span class="k">import</span> <span class="nn">zio.ZIO</span>
<span class="cm">/** Base entry point for a ZIO app, which runs the logic within
* a specified environment
*/</span>
<span class="k">object</span> <span class="nc">Application</span> <span class="k">extends</span> <span class="n">zio</span><span class="o">.</span><span class="nc">App</span> <span class="o">{</span>
<span class="c1">// combine the configuration layer and the standard environment into
</span> <span class="c1">// a new layer.
</span> <span class="k">val</span> <span class="n">defaultLayer</span> <span class="k">=</span> <span class="nc">Configuration</span><span class="o">.</span><span class="n">live</span> <span class="o">++</span> <span class="nc">ZEnv</span><span class="o">.</span><span class="n">live</span>
<span class="c1">// the complete application layer also consists out of the http4server, so
</span> <span class="c1">// add that to the default layers list
</span> <span class="k">val</span> <span class="n">applicationLayer</span> <span class="k">=</span> <span class="n">defaultLayer</span> <span class="o">>>></span> <span class="n">http4sServer</span><span class="o">.</span><span class="nc">Http4sServer</span><span class="o">.</span><span class="n">live</span>
<span class="c1">// now run the application, by providing it with the application layer
</span> <span class="k">override</span> <span class="k">def</span> <span class="n">run</span><span class="o">(</span><span class="n">args</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">String</span><span class="o">])</span><span class="k">:</span> <span class="kt">URIO</span><span class="o">[</span><span class="kt">ZEnv</span>, <span class="kt">ExitCode</span><span class="o">]</span> <span class="k">=</span>
<span class="n">myAppLogic</span><span class="o">.</span><span class="n">provideLayer</span><span class="o">(</span><span class="n">applicationLayer</span><span class="o">).</span><span class="n">exitCode</span>
<span class="c1">// we want a service, but return never
</span> <span class="k">val</span> <span class="n">myAppLogic</span><span class="k">:</span> <span class="kt">URIO</span><span class="o">[</span><span class="kt">http4sServer.Http4sServer.Service</span>, <span class="kt">Unit</span><span class="o">]</span> <span class="k">=</span> <span class="nc">ZIO</span><span class="o">.</span><span class="n">never</span>
<span class="o">}</span>
</code></pre></div></div>
<p>We first combine the <code class="highlighter-rouge">ZEnv.live</code> and <code class="highlighter-rouge">Configuration.live</code> layers into a first layer using the <code class="highlighter-rouge">++</code> operator. Next we pass these dependencies to the http4sServer layer. The <code class="highlighter-rouge">>>></code> operator uses the provided left hand operand as dependencies and returns the layer on the right hand side. This final layer is passed into the application using <code class="highlighter-rouge">provideLayer</code>. Our application is just a <code class="highlighter-rouge">ZIO.never</code>, which means that our application will keep running until interrupted.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[info] running example.Application
[zio-default-async-4] INFO org.http4s.blaze.channel.nio1.NIO1SocketServerGroup - Service bound to address /127.0.0.1:8081
[zio-default-async-4] INFO org.http4s.server.blaze.BlazeServerBuilder -
_ _ _ _ _
| |_| |_| |_ _ __| | | ___
| ' \ _| _| '_ \_ _(_-<
|_||_\__|\__| .__/ |_|/__/
|_|
[zio-default-async-4] INFO org.http4s.server.blaze.BlazeServerBuilder - http4s v1.0.0-M4 on blaze v0.14.13 started at http://127.0.0.1:8081/
</code></pre></div></div>
<h2 id="summary">Summary</h2>
<p>So that’s it for now. We’ve only scratched the surface of ZIO, but this should at least give you a nice idea on how the layering and dependency management works. If I don’t get distracted by something else, I’ll try and post a couple of other articles about ZIO.</p>Jos DirksenI’ve done a lot of Kotlin for the last two years, and have mainly followed Scala and done some pet projects. Since a lot of interesting stuff has been going on in the Scala space for the last couple of months, I want to start writing some more about it, and my experiences with it. So expect some articles on Scala 3 and ZIO. For the first of these set of articles, I’d like to focus a bit on ZIO. I’m exploring the documentation, watched some presentations, but nothing works as best, as just writing a simple application with it. So in this first part on ZIO, I’m going to look at a simple layered ZIO application that (for now) consists out of the following: Loading Configuration: How to load configuration (PureConfig), and provide that as a ZLayer to other services. REST endpoint with HTTP4s: I’ll add a very simple REST layer (only the basics for now), which uses information from the configuration to start a server on a specific host and port. In the end we should have a simple application that starts an HTTP Server, loads configuration in a safe way, and provides us with a simple application that we can build upon for future articles, when we explore more features of ZIO. ZIO Project setup If you want to play along, everything should work out of the box with Metals: Before we can do anything, we need to define the dependencies (build.sbt). When I started playing around with this, there were some dependencies issues between the version of cats, the version of HTTP4S and the zio-cats-interop stuff, so after some checking, the following is a good starting point (I’ll have a look and update this to a newer version in the future). import Dependencies._ ThisBuild / scalaVersion := "2.13.4" ThisBuild / version := "0.1.0-SNAPSHOT" ThisBuild / organization := "com.example" ThisBuild / organizationName := "example" // need to use an older version, since the newest version of http4s // supports the latest cats version, while the zio-interop-cats one // doesn't support this yet. val Http4sVersion = "1.0.0-M4" lazy val root = (project in file(".")) .settings( name := "zio-playground", libraryDependencies += scalaTest % Test, libraryDependencies ++= Seq( "org.http4s" %% "http4s-blaze-server" % Http4sVersion, "org.http4s" %% "http4s-dsl" % Http4sVersion, "dev.zio" %% "zio" % "1.0.6", "dev.zio" %% "zio-streams" % "1.0.6", "dev.zio" %% "zio-interop-cats" % "2.4.0.0", "com.github.pureconfig" %% "pureconfig" % "0.15.0", "org.slf4j" % "slf4j-api" % "1.7.5", "org.slf4j" % "slf4j-simple" % "1.7.5" "dev.zio" %% "zio-test" % "1.0.6" % "test", "dev.zio" %% "zio-test-sbt" % "1.0.6" % "test" ) ) Looking at the dependencies above, nothing out of the ordinary. Note the last two dependencies which we’ll use for testing part of the application. Providing a configuration For the configuration part we’re going to use (PureConfig), which is a typesafe, secure way of loading configurations. How this library works is a bit out of scope, but basically what you do is, you define a set of case classes defining the configuration model, and fill them by loading a configuration file. For this part we have a very basic configuration file: api-config { endpoint = "localhost" port = 8081 } To load this configuration we call ConfigSource.default.load[Config], this returns us with an Either where we have a list of configuration exceptions, or the loaded configuration. To wrap this in ZIO we need to lift it into a ZIO type. We’ll use the Task type from ZIO for this. A Task is one of the default type aliases: type IO[+E, +A] = ZIO[Any, E, A] // Succeed with an `A`, may fail with `E` , no requirements. type Task[+A] = ZIO[Any, Throwable, A] // Succeed with an `A`, may fail with `Throwable`, no requirements. type RIO[-R, +A] = ZIO[R, Throwable, A] // Succeed with an `A`, may fail with `Throwable`, requires an `R`. type UIO[+A] = ZIO[Any, Nothing, A] // Succeed with an `A`, cannot fail , no requirements. type URIO[-R, +A] = ZIO[R, Nothing, A] // Succeed with an `A`, cannot fail , requires an `R`. So we’ve got an operation that can either fail with a Throwable or succeed with an A. To lift our Either we just call fromEither: Task.fromEither( ConfigSource.default .load[Config] .left .map(pureconfig.error.ConfigReaderException.apply) ) Since the load returns a list of ConfigReaderFailures, we just map the left part into an exception, before we can convert it into a ZIO Task. Now that we’ve got this configuration, we can use it in for-comprehensions, pass it to other components that need it and anything else you want to do with it. For this example we’re going one step further, and we’ll define the loading of the configuration as a service, which we can define as a dependency for other services. For ZIO the standard way of doing this is by using ZLayer. I found the documentation a bit hard to follow, but once you start playing around with it, it becomes a lot easier to understand. Below I’ve put the complete source for the configuration service, with inline comments explaining the various steps. Note that this structure is the proposed convention from ZIO, so it’s something you’ll see coming back in multiple sources. import zio.Task import zio.Has import zio.ZIO import pureconfig.ConfigSource import pureconfig.generic.auto._ import zio.ZLayer import zio.Layer // define the domain model used for configuration case class Config(apiConfig: ApiConfig) case class ApiConfig(endpoint: String, port: Int) // define the configuration dependency to inject into the environment // and the related functions object configuration { // Custom type to inject type Configuration = zio.Has[Configuration.Service] // contains the service definition, and multiple implementations // based on the environment we want to run in. object Configuration { // we just have the load function inside the service to load // some configuration trait Service { val load: Task[Config] } // we can have multiple implementations of this service. This is the // live implementation, which loads the config through pureconfig val live: Layer[Nothing, Configuration] = ZLayer.succeed { new Service { override val load: Task[Config] = Task.fromEither( ConfigSource.default .load[Config] .left .map(pureconfig.error.ConfigReaderException.apply) ) } } // helper function which access the correct resource from our environment. val load: ZIO[Configuration, Throwable, Config] = ZIO.accessM(_.get.load) } } What we do here is first define the case classes that map to the configuration file, that is just something specific to pure-config. Next we define the configuration object, where we define the ZIO / ZLayer specific stuff. We define a Configuration type, which is the type we can have other services depend on. This is the type which uniquely identifies that functionality offered by this service. To define this we use zio.Has[A], which following the scaladoc, does the following: The trait Has[A] is used with ZIO environment to express an effect’s dependency on a service of type A. For example, RIO[Has[Console.Service], Unit] is an effect that requires a Console.Service service. Inside the ZIO library, type aliases are provided as shorthands for common services, e.g.: After the type definition, we define a trait with the specific functions provided by this service. In our case this is just the load function (we’d actually could do better by loading during service creation, and assign it to a lazy val or use memomize). The next step we need to do is define the different implementation we have of our service. We could have specific ones for testing, for specific environments and so on. In this case we have one, for which it is the convention to call that one live: val live: Layer[Nothing, Configuration] = ZLayer.succeed { new Service { override val load: Task[Config] = Task.fromEither( ConfigSource.default .load[Config] .left .map(pureconfig.error.ConfigReaderException.apply) ) } } Here we load the configuration, and lift it into the Task. We create an anonymous instance, and return that as the result of the ZLayer.succeed function. ZLayer.succeed means that service creation of this instance will always succeed. In our case that is true, since we only load the config at the point it is needed. There are other ZLayer functions which allow you to create managed resources, or where the creation of the service returns an effect (e.g a Task). For now though, since nothing can go wrong here, we use ZLayer.succeed. The final part of this service, is that we provide a convenience method called load, which mirrors the load function from our service: val load: ZIO[Configuration, Throwable, Config] = ZIO.accessM(_.get.load) This means that if there is a Configuration in the provided environment, we can use this helper function to directly access the load function, and load the configuration. We’ll show you how this is used further down in this article, when we start tying services together. Testing a configuration So far we’ve seen quite some code, but how do we now that this will do what it needs to do. For this ZIO provides some testing libraries, where we can test whether the service we just created does what it needs to do. So let’s write a test: import org.scalatest.flatspec.AnyFlatSpec import org.scalatest.matchers.should.Matchers import zio.test.DefaultRunnableSpec import zio.test._ import Assertion._ object LiveConfigLoaderSpec extends DefaultRunnableSpec { override def spec: ZSpec[Environment, Failure] = suite("ConfigLoaderSpec")( testM("Load the configuration") { for { config <- configuration.Configuration.load } yield { assert(config.apiConfig.port)(equalTo(8081)) } } ).provideCustomLayer(configuration.Configuration.live) } What we do here is that we call the load function we just defined in the companion object, and based on the passed in layer, we can check whether it works. In this example we pass in the configuration.Configuration.live layer. So now when we call configuration.Configuration.load, we should call the load of the pureconfig based configuration. Running the above test will show something like this: + ConfigLoaderSpec + Load the configuration Ran 1 test in 751 ms: 1 succeeded, 0 ignored, 0 failed At this point we’ve got a simple configuration service, but we haven’t got an application yet. So before we move on the http4s stuff, we’re going to set up a minimal application. Minimal application Once again, I’ll show the annotated code, and then go into some of the details: import zio.ExitCode import zio.URIO import zio.ZEnv import configuration.Configuration import example.services.http4sServer import zio.ZIO /** Base entry point for a ZIO app, which runs the logic within * a specified environment */ object Application extends zio.App { // now run the application, by providing it with the application layer override def run(args: List[String]): URIO[ZEnv, ExitCode] = myAppLogic.provideLayer(Configuration.live).exitCode // we can do more stuff here. For now just load the config, cause why not. val myAppLogic = for { config <- Configuration.load } yield ( println(config) ) } Here we extend from zio.App which gives us the override def run(args: List[String]): URIO[ZEnv, ExitCode] function to implement as entrypoint in the application. What we do in this entry point is call our program, and when we do that we provide a set of layers containing our services. In this example we just provide the layer we just defined, where we can load the configuration. In out program myAppLogic we load the configuration (which is present in the environment of our ZIO type.). The result is very simply that we see this: $ sbt run [info] welcome to sbt 1.5.0 (AdoptOpenJDK Java 11.0.7) [info] loading settings for project zio-playground-build-build from metals.sbt ... [info] loading project definition from /home/jos/dev/git/smartjava/zio-playground/project/project [info] loading settings for project zio-playground-build from metals.sbt ... [info] loading project definition from /home/jos/dev/git/smartjava/zio-playground/project [success] Generated .bloop/zio-playground-build.json [success] Total time: 1 s, completed May 17, 2021, 4:56:17 PM [info] loading settings for project root from build.sbt ... [info] set current project to zio-playground (in build file:/home/jos/dev/git/smartjava/zio-playground/) [info] running example.Application Config(ApiConfig(localhost,8081)) What we did above is pretty much the same as we did in the test. We provide a program to execute, and when we execute that program we pass in a set of services that can be accessed through the environment. Adding the HTTP4S server dependency For the last part of this article, we’ll also add a very simple HTTP4S server that will just return ok (we’ll look at the mapping to case classes later). For the HTTP4S server we want it to use the configuration layer we defined earlier. To get the HTTP4S server running we need to take a couple of extra steps to make it work correctly with ZIO. To create the HTTPServer we use the following function on the BlazeServerBuilder def apply[F[_]](executionContext: ExecutionContext)(implicit F: ConcurrentEffect[F], timer: Timer[F]): BlazeServerBuilder[F] If you look at this signature you can see that, besides an explicit executionContext we also need an implicit Timer[F] and implicit ConcurrentEffect[F], where F is the type we want to work with. Since we’re already working with a Task in the config service, lets do that as well for this part. So we want to use the builder like this: BlazeServerBuilder[Task](someExecutionContext). This won’t work however, since we don’t have the correct implicits in scope, and don’t use the types from Cats. Luckily zio provides an interop library that provides us with the correct implicits. For instance the ConcurrentEffect[Task] and Timer[Task are provided by this implicit function from the CatsEffectInstances class: implicit final def taskEffectInstance[R](implicit runtime: Runtime[R]): effect.ConcurrentEffect[RIO[R, *]] = new CatsConcurrentEffect[R](runtime) implicit final def zioTimer[R <: Clock, E]: effect.Timer[ZIO[R, E, *]] = zioTimer0.asInstanceOf[effect.Timer[ZIO[R, E, *]]] As you can see the taskEffectInstance needs a Runtime as implicit parameter, so we need to make sure that our service that we’re defining has access to the ZIO Runtime. And looking back at the initial apply function for the BlazeServerBuilder we can also see that it needs an executionContext. Luckily an executionContext is also provided by the ZIO Runtime. Now that we now which dependencies we need we can create the service. First we’ll look at the basic structure, and then a bit closer at how to create the HTTP4s server: import org.http4s.server.Server import zio._ import example.configuration.Configuration object http4sServer { type Http4sServer = zio.Has[Http4sServer.Service] object Http4sServer { type Service = Has[Server] val live: ZLayer[ZEnv with Configuration, Throwable, Service] = ... } } The code above defines our server, and the signature for the live instance. As we mentioned we want to use information from the Configuration dependency to configure the service, and we need to have access to the zio runtime to correctly set up HTTP4s. ZIO has helper functions for creating the runtime, so lets look at the implementation of the live instance. val live: ZLayer[ZEnv with Configuration, Throwable, Service] = ZLayer.fromManaged { for { // we should probably cache this, and make this a function, where we // pass in the relevant configuration config <- Configuration.load.toManaged_ // The implicit runtime provided the implicits needed to create the ConcurrentEffect. // This is done by the zio / cats interop imports. server <- ZManaged.runtime[ZEnv].flatMap { implicit runtime: Runtime[ZEnv] => BlazeServerBuilder[Task](runtime.platform.executor.asEC) .bindHttp(config.apiConfig.port, config.apiConfig.endpoint) .withHttpApp(Routes.routes) .resource .toManagedZIO } } yield (server) } Before we look at the fromManaged, let’s first look at the way we create the server. As we mentioned we need access the runtime, ZIO provides a useful helper for this called ZManaged.runtime, where we create a runtime based on the passed in type. Using this runtime, we can create the BlazeServerBuilder. When using cats we would use the resource to safely start and stop the server. In ZIO world this is called a ZManaged: A ZManaged[R, E, A] is a managed resource of type A, which may be used by invoking the use method of the resource. The resource will be automatically acquired before the resource is used, and automatically released after the resource is used. The resource from the call to .resource can be converted to a ZManaged by calling toManagedZIO. Since this returns a ZManaged type, we also need to make sure that the other types in our for-comprehension are also ZManageds. So we use the toManaged_ function to convert the result from the Configuration.load to a ZManaged. At the end of the for-comprehension we can then convert the resulting ZManaged into a ZLayer by calling the fromManaged. This seems like a lot of work, but it nicely shows how well ZIO can play with existing Cats types. Defining a minimal API and running the application Without going too much into detail on the http4s specifics, the following are the routes we support for now (don’t look at the implementation, since it’s just some placeholders for future implementation) object Routes { private val dsl = Http4sDsl[Task] import dsl._ def routes = HttpRoutes .of[Task] { case GET -> Root / "users" / IntVar(id) => { Created("A value here") } case POST -> Root / "users" => { Created("And another one here") } } .orNotFound } And with some small updates our main program looks like this: package example import zio.ExitCode import zio.URIO import zio.ZEnv import configuration.Configuration import example.services.http4sServer import zio.ZIO /** Base entry point for a ZIO app, which runs the logic within * a specified environment */ object Application extends zio.App { // combine the configuration layer and the standard environment into // a new layer. val defaultLayer = Configuration.live ++ ZEnv.live // the complete application layer also consists out of the http4server, so // add that to the default layers list val applicationLayer = defaultLayer >>> http4sServer.Http4sServer.live // now run the application, by providing it with the application layer override def run(args: List[String]): URIO[ZEnv, ExitCode] = myAppLogic.provideLayer(applicationLayer).exitCode // we want a service, but return never val myAppLogic: URIO[http4sServer.Http4sServer.Service, Unit] = ZIO.never } We first combine the ZEnv.live and Configuration.live layers into a first layer using the ++ operator. Next we pass these dependencies to the http4sServer layer. The >>> operator uses the provided left hand operand as dependencies and returns the layer on the right hand side. This final layer is passed into the application using provideLayer. Our application is just a ZIO.never, which means that our application will keep running until interrupted. [info] running example.Application [zio-default-async-4] INFO org.http4s.blaze.channel.nio1.NIO1SocketServerGroup - Service bound to address /127.0.0.1:8081 [zio-default-async-4] INFO org.http4s.server.blaze.BlazeServerBuilder - _ _ _ _ _ | |_| |_| |_ _ __| | | ___ | ' \ _| _| '_ \_ _(_-< |_||_\__|\__| .__/ |_|/__/ |_| [zio-default-async-4] INFO org.http4s.server.blaze.BlazeServerBuilder - http4s v1.0.0-M4 on blaze v0.14.13 started at http://127.0.0.1:8081/ Summary So that’s it for now. We’ve only scratched the surface of ZIO, but this should at least give you a nice idea on how the layering and dependency management works. If I don’t get distracted by something else, I’ll try and post a couple of other articles about ZIO.Running a minimal Kafka instance on K8S2020-03-06T00:00:00+01:002020-03-06T00:00:00+01:00http://www.smartjava.org/content/minimal-kafka-instance-for-k8s%20copy<p>A couple of days ago I ran into an issue with our hosted Kafka environment, and we had to give a demo. So I was looking for a quick way to run a simple ephemeral Kafka instance on our kubernetes cluster. I looked around a bit, and couldn’t find one working out of the box, so I adapted one I found and which almost worked (https://dzone.com/articles/ultimate-guide-to-installing-kafka-docker-on-kuber) which used the standard docker images from <a href="https://hub.docker.com/r/wurstmeister/kafka/">here</a></p>
<p>The complete setup will consist out of:</p>
<ul>
<li>A single deployment for zookeeper. This will create a single zookeeper pod, with the name <code class="highlighter-rouge">zoo1</code></li>
<li>A single deployment for kafka. This will create a single kafka pod, with the name <code class="highlighter-rouge">kafka-broker0</code></li>
<li>two services; one that exposes zookeeper, and one that exposes the kafka-broker</li>
<li>a final deployment that also adds <code class="highlighter-rouge">kafkacat</code> so we can quickly test whether our broker is working and accessible</li>
</ul>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nn">---</span>
<span class="na">apiVersion</span><span class="pi">:</span> <span class="s">v1</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">Service</span>
<span class="na">metadata</span><span class="pi">:</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">zoo1</span>
<span class="na">labels</span><span class="pi">:</span>
<span class="na">app</span><span class="pi">:</span> <span class="s">zookeeper-1</span>
<span class="na">spec</span><span class="pi">:</span>
<span class="na">ports</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">client</span>
<span class="na">port</span><span class="pi">:</span> <span class="s">2181</span>
<span class="na">protocol</span><span class="pi">:</span> <span class="s">TCP</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">follower</span>
<span class="na">port</span><span class="pi">:</span> <span class="s">2888</span>
<span class="na">protocol</span><span class="pi">:</span> <span class="s">TCP</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">leader</span>
<span class="na">port</span><span class="pi">:</span> <span class="s">3888</span>
<span class="na">protocol</span><span class="pi">:</span> <span class="s">TCP</span>
<span class="na">selector</span><span class="pi">:</span>
<span class="na">app</span><span class="pi">:</span> <span class="s">zookeeper-1</span>
<span class="nn">---</span>
<span class="na">apiVersion</span><span class="pi">:</span> <span class="s">v1</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">Service</span>
<span class="na">metadata</span><span class="pi">:</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">kafka-service</span>
<span class="na">labels</span><span class="pi">:</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">kafka</span>
<span class="na">spec</span><span class="pi">:</span>
<span class="na">ports</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">port</span><span class="pi">:</span> <span class="s">9092</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">kafka-port</span>
<span class="na">protocol</span><span class="pi">:</span> <span class="s">TCP</span>
<span class="na">selector</span><span class="pi">:</span>
<span class="na">app</span><span class="pi">:</span> <span class="s">kafka</span>
<span class="na">id</span><span class="pi">:</span> <span class="s2">"</span><span class="s">0"</span>
<span class="nn">---</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">Deployment</span>
<span class="na">apiVersion</span><span class="pi">:</span> <span class="s">extensions/v1beta1</span>
<span class="na">metadata</span><span class="pi">:</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">zookeeper-deployment-1</span>
<span class="na">spec</span><span class="pi">:</span>
<span class="na">template</span><span class="pi">:</span>
<span class="na">metadata</span><span class="pi">:</span>
<span class="na">labels</span><span class="pi">:</span>
<span class="na">app</span><span class="pi">:</span> <span class="s">zookeeper-1</span>
<span class="na">spec</span><span class="pi">:</span>
<span class="na">containers</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">zoo1</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">digitalwonderland/zookeeper</span>
<span class="na">ports</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">containerPort</span><span class="pi">:</span> <span class="s">2181</span>
<span class="na">env</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">ZOOKEEPER_ID</span>
<span class="na">value</span><span class="pi">:</span> <span class="s2">"</span><span class="s">1"</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">ZOOKEEPER_SERVER_1</span>
<span class="na">value</span><span class="pi">:</span> <span class="s">zoo1</span>
<span class="nn">---</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">Deployment</span>
<span class="na">apiVersion</span><span class="pi">:</span> <span class="s">extensions/v1beta1</span>
<span class="na">metadata</span><span class="pi">:</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">kafka-broker0</span>
<span class="na">spec</span><span class="pi">:</span>
<span class="na">template</span><span class="pi">:</span>
<span class="na">metadata</span><span class="pi">:</span>
<span class="na">labels</span><span class="pi">:</span>
<span class="na">app</span><span class="pi">:</span> <span class="s">kafka</span>
<span class="na">id</span><span class="pi">:</span> <span class="s2">"</span><span class="s">0"</span>
<span class="na">spec</span><span class="pi">:</span>
<span class="na">containers</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">kafka</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">wurstmeister/kafka</span>
<span class="na">ports</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">containerPort</span><span class="pi">:</span> <span class="s">9092</span>
<span class="na">env</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">KAFKA_ADVERTISED_PORT</span>
<span class="na">value</span><span class="pi">:</span> <span class="s2">"</span><span class="s">9092"</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">KAFKA_ADVERTISED_HOST_NAME</span>
<span class="na">value</span><span class="pi">:</span> <span class="s">kafka-service</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">KAFKA_ZOOKEEPER_CONNECT</span>
<span class="na">value</span><span class="pi">:</span> <span class="s">zoo1:2181</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">KAFKA_BROKER_ID</span>
<span class="na">value</span><span class="pi">:</span> <span class="s2">"</span><span class="s">0"</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">KAFKA_CREATE_TOPICS</span>
<span class="na">value</span><span class="pi">:</span> <span class="s">sample.topic:1:1</span>
<span class="nn">---</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">Deployment</span>
<span class="na">apiVersion</span><span class="pi">:</span> <span class="s">extensions/v1beta1</span>
<span class="na">metadata</span><span class="pi">:</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">kafka-cat</span>
<span class="na">spec</span><span class="pi">:</span>
<span class="na">template</span><span class="pi">:</span>
<span class="na">metadata</span><span class="pi">:</span>
<span class="na">labels</span><span class="pi">:</span>
<span class="na">app</span><span class="pi">:</span> <span class="s">kafka-cat</span>
<span class="na">spec</span><span class="pi">:</span>
<span class="na">containers</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">kafka-cat</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">confluentinc/cp-kafkacat</span>
<span class="na">command</span><span class="pi">:</span> <span class="pi">[</span><span class="s2">"</span><span class="s">/bin/sh"</span><span class="pi">]</span>
<span class="na">args</span><span class="pi">:</span> <span class="pi">[</span><span class="s2">"</span><span class="s">-c"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">trap</span><span class="nv"> </span><span class="s">:</span><span class="nv"> </span><span class="s">TERM</span><span class="nv"> </span><span class="s">INT;</span><span class="nv"> </span><span class="s">sleep</span><span class="nv"> </span><span class="s">infinity</span><span class="nv"> </span><span class="s">&</span><span class="nv"> </span><span class="s">wait"</span><span class="pi">]</span>
</code></pre></div></div>
<p>To deploy it simple save this configuration as <code class="highlighter-rouge">kubernetes-kafka-ephemeral.yml</code>:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>kubectl apply <span class="nt">-f</span> ./kubernetes-kafka-ephemeral.yml
service/zoo1 created
service/kafka-service created
deployment.extensions/zookeeper-deployment-1 created
deployment.extensions/kafka-broker0 created
deployment.extensions/kafka-cat created
</code></pre></div></div>
<p>And looking at the resources you’ll see that the correct items have been created (the kafka broker might restart if zookeeper isn’t up yet, but that’ll resolve itself):</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="nv">$ </span>kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT<span class="o">(</span>S<span class="o">)</span> AGE
kafka-service ClusterIP 10.105.141.140 <none> 9092/TCP 72s
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 49m
zoo1 ClusterIP 10.108.158.62 <none> 2181/TCP,2888/TCP,3888/TCP 72s
kubectl get pods
NAME READY STATUS RESTARTS AGE
kafka-broker0-677f5b7ccf-gsstk 1/1 Running 0 76s
kafka-cat-84b9cfdf45-x9wk7 1/1 Running 0 76s
zookeeper-deployment-1-684478678c-d8m5m 1/1 Running 0 76s
<span class="nv">$ </span>kubectl get deployments
NAME READY UP-TO-DATE AVAILABLE AGE
kafka-broker0 1/1 1 1 80s
kafka-cat 1/1 1 1 80s
zookeeper-deployment-1 1/1 1 1 80s
</code></pre></div></div>
<p><code class="highlighter-rouge">kafkacat</code> runs in a pod which won’t exit, so you can just <code class="highlighter-rouge">exec</code> into the pod, and see if everything works. Easiest way to test is to just
open two shells and run the kafkacat commands (<code class="highlighter-rouge">kafkacat -P -b kafka-service:9092 -t test</code> and <code class="highlighter-rouge">kafkacat -b kafka-service:9092 -t test</code>):</p>
<p><img src="../../images/kafkacat.gif" alt="Kafkacat in action" /></p>Jos DirksenA couple of days ago I ran into an issue with our hosted Kafka environment, and we had to give a demo. So I was looking for a quick way to run a simple ephemeral Kafka instance on our kubernetes cluster. I looked around a bit, and couldn’t find one working out of the box, so I adapted one I found and which almost worked (https://dzone.com/articles/ultimate-guide-to-installing-kafka-docker-on-kuber) which used the standard docker images from here The complete setup will consist out of: A single deployment for zookeeper. This will create a single zookeeper pod, with the name zoo1 A single deployment for kafka. This will create a single kafka pod, with the name kafka-broker0 two services; one that exposes zookeeper, and one that exposes the kafka-broker a final deployment that also adds kafkacat so we can quickly test whether our broker is working and accessible --- apiVersion: v1 kind: Service metadata: name: zoo1 labels: app: zookeeper-1 spec: ports: - name: client port: 2181 protocol: TCP - name: follower port: 2888 protocol: TCP - name: leader port: 3888 protocol: TCP selector: app: zookeeper-1 --- apiVersion: v1 kind: Service metadata: name: kafka-service labels: name: kafka spec: ports: - port: 9092 name: kafka-port protocol: TCP selector: app: kafka id: "0" --- kind: Deployment apiVersion: extensions/v1beta1 metadata: name: zookeeper-deployment-1 spec: template: metadata: labels: app: zookeeper-1 spec: containers: - name: zoo1 image: digitalwonderland/zookeeper ports: - containerPort: 2181 env: - name: ZOOKEEPER_ID value: "1" - name: ZOOKEEPER_SERVER_1 value: zoo1 --- kind: Deployment apiVersion: extensions/v1beta1 metadata: name: kafka-broker0 spec: template: metadata: labels: app: kafka id: "0" spec: containers: - name: kafka image: wurstmeister/kafka ports: - containerPort: 9092 env: - name: KAFKA_ADVERTISED_PORT value: "9092" - name: KAFKA_ADVERTISED_HOST_NAME value: kafka-service - name: KAFKA_ZOOKEEPER_CONNECT value: zoo1:2181 - name: KAFKA_BROKER_ID value: "0" - name: KAFKA_CREATE_TOPICS value: sample.topic:1:1 --- kind: Deployment apiVersion: extensions/v1beta1 metadata: name: kafka-cat spec: template: metadata: labels: app: kafka-cat spec: containers: - name: kafka-cat image: confluentinc/cp-kafkacat command: ["/bin/sh"] args: ["-c", "trap : TERM INT; sleep infinity & wait"] To deploy it simple save this configuration as kubernetes-kafka-ephemeral.yml: $ kubectl apply -f ./kubernetes-kafka-ephemeral.yml service/zoo1 created service/kafka-service created deployment.extensions/zookeeper-deployment-1 created deployment.extensions/kafka-broker0 created deployment.extensions/kafka-cat created And looking at the resources you’ll see that the correct items have been created (the kafka broker might restart if zookeeper isn’t up yet, but that’ll resolve itself): $ kubectl get services NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kafka-service ClusterIP 10.105.141.140 <none> 9092/TCP 72s kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 49m zoo1 ClusterIP 10.108.158.62 <none> 2181/TCP,2888/TCP,3888/TCP 72s kubectl get pods NAME READY STATUS RESTARTS AGE kafka-broker0-677f5b7ccf-gsstk 1/1 Running 0 76s kafka-cat-84b9cfdf45-x9wk7 1/1 Running 0 76s zookeeper-deployment-1-684478678c-d8m5m 1/1 Running 0 76s $ kubectl get deployments NAME READY UP-TO-DATE AVAILABLE AGE kafka-broker0 1/1 1 1 80s kafka-cat 1/1 1 1 80s zookeeper-deployment-1 1/1 1 1 80s kafkacat runs in a pod which won’t exit, so you can just exec into the pod, and see if everything works. Easiest way to test is to just open two shells and run the kafkacat commands (kafkacat -P -b kafka-service:9092 -t test and kafkacat -b kafka-service:9092 -t test):My 10 goto terminal commands2020-03-01T00:00:00+01:002020-03-01T00:00:00+01:00http://www.smartjava.org/content/my-10-goto-terminal-commands<p>There are a lot of useful terminal/shell commands which can make your live a lot easier. I do a lot of backend development, and there is a set of commands which I use a lot during development, testing or operations work. So maybe this list can introduce you to a command that can make your live a little bit easier.</p>
<h2 id="jq">jq</h2>
<p><code class="highlighter-rouge">jq</code> is a command which can be used to parse and transform json. I use it a lot in scripts to parse and extract information from APIs, or to quickly format a JSON file. The basic syntax is very easy, but it provides a very large set of operations and command to manipulate the JSON.</p>
<p><img src="../../images/sj-jq-2.gif" alt="JQ for parsing JSON" /></p>
<p>You can find more information about this tool here: <a href="https://github.com/stedolan/jq">JQ on GitHub</a></p>
<h2 id="autojump">Autojump</h2>
<p>A little known command <code class="highlighter-rouge">autojump</code>, which keeps tracks of your most used directories, so you can quickly jump between directories. There is a convenience wrapper called <code class="highlighter-rouge">j</code>, so I hardly have to use <code class="highlighter-rouge">cd</code> to jump between directories anymore, and can just use <code class="highlighter-rouge">j</code> instead.</p>
<p><img src="../../images/sj-autojump-2.gif" alt="Autojump" /></p>
<p>You can find more information about this tool here: <a href="https://github.com/wting/autojump">Autojump on GitHub</a></p>
<h2 id="date">Date</h2>
<p>Often during work I need to quickly get the current unix epoch timestamp, an ISO formatted date time or quickly determine what time a timestamp actually is. There are a number of websites out there that do this, but I usually just use the specific date commands (and since they are in my history, I usually don’t have to keep the exact parameters in mind). I often use the following:</p>
<ul>
<li><code class="highlighter-rouge">date +%s</code>: Print out the current unix epoch timestamp in seconds</li>
<li><code class="highlighter-rouge">date -u +"%Y-%m-%dT%H:%M:%SZ"</code>: Print out an iso-8601 timestamp as UTC</li>
<li>or <code class="highlighter-rouge">date --iso-8601=seconds</code>: Print out an iso-8601 timestamp using local timezone</li>
<li><code class="highlighter-rouge">date -d@1550841014</code>: Convert epoch seconds back to a readable date</li>
</ul>
<p><img src="../../images/sj-date-2.gif" alt="Date" /></p>
<h2 id="uuid-gen">uuid-Gen</h2>
<p>At my last couple of projects we used uuids as unique ids for entities. While this works great, it sometimes gets annoying when writing tests or preparing some data. Luckily we can just use another command line tool to quickly generate uuids.</p>
<ul>
<li><code class="highlighter-rouge">uuidgen</code>: Just creates a uuid with all uppercase characters. Not all tools though work correctly with upper case characters for the UUID strings.</li>
<li><code class="highlighter-rouge">uuidgen | tr "[:upper:]" "[:lower:]"</code>: Does the same as the previous command, but this time generates them as lower case characters.</li>
</ul>
<p><img src="../../images/sj-uuidgen-2.gif" alt="Uuid-gen" /></p>
<h2 id="generate-passwords">generate passwords</h2>
<p>Another tool which is regularly need, especially when generating users, is a simple password generator. You can use online tools for this, or tools like lastpass. Usually, though, it is quicker to just open a terminal and hit the following command a couple of times: <code class="highlighter-rouge">openssl rand -base64 12</code>:</p>
<p><img src="../../images/sj-pwd-2.gif" alt="openssl" /></p>
<h2 id="ncal">ncal</h2>
<p>Often I need to have quick access to a calendar. Either to plan something, look at what week it is, or just to determine on which day a specific date falls. For an actual agenda, I use google calendar, but I don’t like having to open gmail, or the Mac calendar application, for these simple things. Luckily there are of course CLI apps for that.</p>
<p><img src="../../images/sj-ncal-2.gif" alt="ncal" /></p>
<p>And this is of course a great reason to use <code class="highlighter-rouge">lolcat</code>. 😊</p>
<h2 id="hub">hub</h2>
<p>Most developers use <code class="highlighter-rouge">git</code> to store their data, and often also work with github to store their code. While GitHub works great for most work, and with the normal <code class="highlighter-rouge">git</code> command you can do most of the daily jobs, there are some things missing. The biggest thing I miss when working with gitHub, though, is that you can’t simply create or manage PRs from the command line. You have to open the UI, and do it from there. Luckily, though, there is a command called <code class="highlighter-rouge">hub</code> which fills in this missing gap.</p>
<p><img src="../../images/sj-hub-2.gif" alt="hub" /></p>
<p>More information about hub can be found here: <a href="https://hub.github.com/">https://hub.github.com/</a></p>
<h2 id="curl">curl</h2>
<p>This command is one we’ve already seen in the beginning of this article when we were discussing <code class="highlighter-rouge">jq</code>, and one which most people will already know. I still list it, since I use it so often, especially in combination with browsers like Firefox and Chrome allowing you to copy network requests as <code class="highlighter-rouge">curl</code> commands, makes this a very useful tool. I often use it together with <a href="https://insomnia.rest/">Insomnia</a>, which I use as a general REST client, and which also allows you to copy specific requests as curl commands.</p>
<p><img src="../../images/sj-curl-2.gif" alt="hub" /></p>
<h2 id="htop">htop</h2>
<p>I develop on a mac, and sometimes need to determine why my system is slowing down to a crawl (answer: usually IntelliJ Idea), or when I’m running a large set of services, I need to monitor what is happening. For this you could use normal <code class="highlighter-rouge">top</code>, the Activity Monitor, or just <code class="highlighter-rouge">ps</code>, but I often have a simple terminal window open with <code class="highlighter-rouge">htop</code>:</p>
<p><img src="../../images/sj-htop-2.gif" alt="hub" /></p>
<h1 id="vi">vi</h1>
<p>You either love or hate <code class="highlighter-rouge">vi(m)</code>, and I tend to really like it. It’s present on most systems, once you’re used to it, provides a lot of quick commands for editing text files, and is often defined as the standard editor for git. There are thousands of extensions to customize it, use it as a complete IDE, or create an enhanced editor using <a href="https://spacevim.org/">SpaceVim</a></p>
<p><img src="../../images/sj-vi-2.gif" alt="vi" /></p>
<h2 id="cd---git-checkout--">cd -, git checkout -</h2>
<p>The last command I wanted to show isn’t a real command, but more of a way of navigating directories and branches. I had been doing linux and mac for a couple of years before I learned about this, and it really makes switching directories and switches branches a lot easier and quicker. All you need to remember is that you can quickly switch to the previous directory by just doing <code class="highlighter-rouge">cd -</code>, and switching to the previous <code class="highlighter-rouge">git</code> branch by doing <code class="highlighter-rouge">git checkout -</code>.</p>
<p><img src="../../images/sj-cd-2.gif" alt="cd -" /></p>
<h2 id="honorable-mention-terminalizer">Honorable mention: terminalizer</h2>
<p>As an honorable mention I’d like to add <code class="highlighter-rouge">terminalizer</code>, with this tool (get it from <a href="https://github.com/faressoft/terminalizer">here</a>), you can very quickly capture a terminals input and output, and generate the Gifs you see in this page. You can even use it to capture <code class="highlighter-rouge">terminalizer</code> itself in action.</p>
<p><img src="../../images/sj-term-2.gif" alt="terminalizer" /></p>Jos DirksenThere are a lot of useful terminal/shell commands which can make your live a lot easier. I do a lot of backend development, and there is a set of commands which I use a lot during development, testing or operations work. So maybe this list can introduce you to a command that can make your live a little bit easier. jq jq is a command which can be used to parse and transform json. I use it a lot in scripts to parse and extract information from APIs, or to quickly format a JSON file. The basic syntax is very easy, but it provides a very large set of operations and command to manipulate the JSON. You can find more information about this tool here: JQ on GitHub Autojump A little known command autojump, which keeps tracks of your most used directories, so you can quickly jump between directories. There is a convenience wrapper called j, so I hardly have to use cd to jump between directories anymore, and can just use j instead. You can find more information about this tool here: Autojump on GitHub Date Often during work I need to quickly get the current unix epoch timestamp, an ISO formatted date time or quickly determine what time a timestamp actually is. There are a number of websites out there that do this, but I usually just use the specific date commands (and since they are in my history, I usually don’t have to keep the exact parameters in mind). I often use the following: date +%s: Print out the current unix epoch timestamp in seconds date -u +"%Y-%m-%dT%H:%M:%SZ": Print out an iso-8601 timestamp as UTC or date --iso-8601=seconds: Print out an iso-8601 timestamp using local timezone date -d@1550841014: Convert epoch seconds back to a readable date uuid-Gen At my last couple of projects we used uuids as unique ids for entities. While this works great, it sometimes gets annoying when writing tests or preparing some data. Luckily we can just use another command line tool to quickly generate uuids. uuidgen: Just creates a uuid with all uppercase characters. Not all tools though work correctly with upper case characters for the UUID strings. uuidgen | tr "[:upper:]" "[:lower:]": Does the same as the previous command, but this time generates them as lower case characters. generate passwords Another tool which is regularly need, especially when generating users, is a simple password generator. You can use online tools for this, or tools like lastpass. Usually, though, it is quicker to just open a terminal and hit the following command a couple of times: openssl rand -base64 12: ncal Often I need to have quick access to a calendar. Either to plan something, look at what week it is, or just to determine on which day a specific date falls. For an actual agenda, I use google calendar, but I don’t like having to open gmail, or the Mac calendar application, for these simple things. Luckily there are of course CLI apps for that. And this is of course a great reason to use lolcat. 😊 hub Most developers use git to store their data, and often also work with github to store their code. While GitHub works great for most work, and with the normal git command you can do most of the daily jobs, there are some things missing. The biggest thing I miss when working with gitHub, though, is that you can’t simply create or manage PRs from the command line. You have to open the UI, and do it from there. Luckily, though, there is a command called hub which fills in this missing gap. More information about hub can be found here: https://hub.github.com/ curl This command is one we’ve already seen in the beginning of this article when we were discussing jq, and one which most people will already know. I still list it, since I use it so often, especially in combination with browsers like Firefox and Chrome allowing you to copy network requests as curl commands, makes this a very useful tool. I often use it together with Insomnia, which I use as a general REST client, and which also allows you to copy specific requests as curl commands. htop I develop on a mac, and sometimes need to determine why my system is slowing down to a crawl (answer: usually IntelliJ Idea), or when I’m running a large set of services, I need to monitor what is happening. For this you could use normal top, the Activity Monitor, or just ps, but I often have a simple terminal window open with htop: vi You either love or hate vi(m), and I tend to really like it. It’s present on most systems, once you’re used to it, provides a lot of quick commands for editing text files, and is often defined as the standard editor for git. There are thousands of extensions to customize it, use it as a complete IDE, or create an enhanced editor using SpaceVim cd -, git checkout - The last command I wanted to show isn’t a real command, but more of a way of navigating directories and branches. I had been doing linux and mac for a couple of years before I learned about this, and it really makes switching directories and switches branches a lot easier and quicker. All you need to remember is that you can quickly switch to the previous directory by just doing cd -, and switching to the previous git branch by doing git checkout -. Honorable mention: terminalizer As an honorable mention I’d like to add terminalizer, with this tool (get it from here), you can very quickly capture a terminals input and output, and generate the Gifs you see in this page. You can even use it to capture terminalizer itself in action.Standalone Apollo Graphql Websocket Subscriptions2020-02-28T00:00:00+01:002020-02-28T00:00:00+01:00http://www.smartjava.org/content/standalone-apollo-client<p>This is just a quick article on how to create a simple standalone Apollo client to test Graphql subscriptions. I needed
something like this when we ran into some strange issues at work, where our Graphql backend (Kotlin) and Apollo React frontend
stopped working, or got into a refresh loop. To easily test this I wanted something I could script to easily run a couple of
scenarios without having to use a browser.</p>
<p>I couldn’t really find a good example, though, how to do this, so after looking through some samples, and libraries, I came up with this:</p>
<div class="language-typescript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">import</span> <span class="p">{</span> <span class="nx">ApolloClient</span> <span class="p">}</span> <span class="k">from</span> <span class="s1">'apollo-client'</span><span class="p">;</span>
<span class="k">import</span> <span class="p">{</span> <span class="nx">InMemoryCache</span> <span class="p">}</span> <span class="k">from</span> <span class="s1">'apollo-cache-inmemory'</span><span class="p">;</span>
<span class="k">import</span> <span class="nx">gql</span> <span class="k">from</span> <span class="s1">'graphql-tag'</span><span class="p">;</span>
<span class="k">import</span> <span class="p">{</span> <span class="nx">WebSocketLink</span> <span class="p">}</span> <span class="k">from</span> <span class="s2">"apollo-link-ws"</span><span class="p">;</span>
<span class="k">import</span> <span class="p">{</span> <span class="nx">SubscriptionClient</span> <span class="p">}</span> <span class="k">from</span> <span class="s2">"subscriptions-transport-ws"</span><span class="p">;</span>
<span class="k">import</span> <span class="nx">ws</span> <span class="k">from</span> <span class="s1">'ws'</span><span class="p">;</span>
<span class="kd">let</span> <span class="nx">args</span> <span class="o">=</span> <span class="nx">process</span><span class="p">.</span><span class="nx">argv</span><span class="p">.</span><span class="nx">slice</span><span class="p">(</span><span class="mi">2</span><span class="p">);</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">args</span><span class="p">.</span><span class="nx">length</span> <span class="o">!=</span> <span class="mi">1</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="s2">`Expected a single arguments </span><span class="p">${</span><span class="nx">process</span><span class="p">.</span><span class="nx">argv</span><span class="p">}</span><span class="s2"> as query`</span><span class="p">)</span>
<span class="nx">process</span><span class="p">.</span><span class="nx">exit</span><span class="p">(</span><span class="mi">1</span><span class="p">);</span>
<span class="p">}</span>
<span class="kd">let</span> <span class="nx">query</span> <span class="o">=</span> <span class="nx">args</span><span class="p">[</span><span class="mi">0</span><span class="p">];</span>
<span class="c1">// the endpoint we're connecting to</span>
<span class="kd">const</span> <span class="nx">GRAPHQL_ENDPOINT</span> <span class="o">=</span> <span class="s2">"wss://some/endpoint/for/subscriptions"</span><span class="p">;</span>
<span class="c1">// setup a subscription client for that endpoint</span>
<span class="kd">const</span> <span class="nx">subscriptionClient</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">SubscriptionClient</span><span class="p">(</span><span class="nx">GRAPHQL_ENDPOINT</span><span class="p">,</span> <span class="p">{</span>
<span class="na">reconnect</span><span class="p">:</span> <span class="kc">true</span>
<span class="p">},</span> <span class="nx">ws</span><span class="p">);</span>
<span class="c1">// using the subscription client to set up a websocket link</span>
<span class="c1">// which can be passed into the apollo client</span>
<span class="kd">const</span> <span class="nx">link</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">WebSocketLink</span><span class="p">(</span><span class="nx">subscriptionClient</span><span class="p">);</span>
<span class="kd">let</span> <span class="nx">cc</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">ApolloClient</span><span class="p">({</span>
<span class="na">link</span><span class="p">:</span> <span class="nx">link</span><span class="p">,</span>
<span class="na">cache</span><span class="p">:</span> <span class="k">new</span> <span class="nx">InMemoryCache</span><span class="p">()</span>
<span class="p">});</span>
<span class="c1">// The subscription we want to follow, just parse it from the string</span>
<span class="kd">const</span> <span class="nx">asGql</span> <span class="o">=</span> <span class="nx">gql</span><span class="s2">`</span><span class="p">${</span><span class="nx">query</span><span class="p">}</span><span class="s2">`</span>
<span class="c1">// subscribe</span>
<span class="kd">const</span> <span class="nx">s</span> <span class="o">=</span> <span class="nx">cc</span><span class="p">.</span><span class="nx">subscribe</span><span class="p">({</span>
<span class="na">query</span><span class="p">:</span> <span class="nx">asGql</span>
<span class="p">})</span>
<span class="nx">s</span><span class="p">.</span><span class="nx">subscribe</span><span class="p">({</span>
<span class="na">next</span><span class="p">:</span> <span class="p">({</span> <span class="nx">data</span> <span class="p">})</span> <span class="o">=></span> <span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="nx">data</span><span class="p">)</span>
<span class="p">});</span>
</code></pre></div></div>
<p>And the following <code class="highlighter-rouge">package.json</code>:</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="s2">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"standalone-apollo-client"</span><span class="p">,</span><span class="w">
</span><span class="s2">"version"</span><span class="p">:</span><span class="w"> </span><span class="s2">"1.0.0"</span><span class="p">,</span><span class="w">
</span><span class="s2">"description"</span><span class="p">:</span><span class="w"> </span><span class="s2">""</span><span class="p">,</span><span class="w">
</span><span class="s2">"main"</span><span class="p">:</span><span class="w"> </span><span class="s2">"index.js"</span><span class="p">,</span><span class="w">
</span><span class="s2">"author"</span><span class="p">:</span><span class="w"> </span><span class="s2">""</span><span class="p">,</span><span class="w">
</span><span class="s2">"license"</span><span class="p">:</span><span class="w"> </span><span class="s2">"ISC"</span><span class="p">,</span><span class="w">
</span><span class="s2">"devDependencies"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"typescript"</span><span class="p">:</span><span class="w"> </span><span class="s2">"^3.8.2"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"dependencies"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"@oitmain/apollo-client-standalone"</span><span class="p">:</span><span class="w"> </span><span class="s2">"^1.0.0"</span><span class="p">,</span><span class="w">
</span><span class="s2">"apollo-link-ws"</span><span class="p">:</span><span class="w"> </span><span class="s2">"^1.0.19"</span><span class="p">,</span><span class="w">
</span><span class="s2">"subscriptions-transport-ws"</span><span class="p">:</span><span class="w"> </span><span class="s2">"^0.9.16"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>This small node script takes a graphql subscription as its argument, and by just changing the callback function, you can quickly test your subscriptions. In the end it really helped me determining what was wrong (which in the end wasn’t in the FE or the BE, with an obscure HAProxy configuration).</p>Jos DirksenThis is just a quick article on how to create a simple standalone Apollo client to test Graphql subscriptions. I needed something like this when we ran into some strange issues at work, where our Graphql backend (Kotlin) and Apollo React frontend stopped working, or got into a refresh loop. To easily test this I wanted something I could script to easily run a couple of scenarios without having to use a browser. I couldn’t really find a good example, though, how to do this, so after looking through some samples, and libraries, I came up with this: import { ApolloClient } from 'apollo-client'; import { InMemoryCache } from 'apollo-cache-inmemory'; import gql from 'graphql-tag'; import { WebSocketLink } from "apollo-link-ws"; import { SubscriptionClient } from "subscriptions-transport-ws"; import ws from 'ws'; let args = process.argv.slice(2); if (args.length != 1) { console.log(`Expected a single arguments ${process.argv} as query`) process.exit(1); } let query = args[0]; // the endpoint we're connecting to const GRAPHQL_ENDPOINT = "wss://some/endpoint/for/subscriptions"; // setup a subscription client for that endpoint const subscriptionClient = new SubscriptionClient(GRAPHQL_ENDPOINT, { reconnect: true }, ws); // using the subscription client to set up a websocket link // which can be passed into the apollo client const link = new WebSocketLink(subscriptionClient); let cc = new ApolloClient({ link: link, cache: new InMemoryCache() }); // The subscription we want to follow, just parse it from the string const asGql = gql`${query}` // subscribe const s = cc.subscribe({ query: asGql }) s.subscribe({ next: ({ data }) => console.log(data) }); And the following package.json: { "name": "standalone-apollo-client", "version": "1.0.0", "description": "", "main": "index.js", "author": "", "license": "ISC", "devDependencies": { "typescript": "^3.8.2" }, "dependencies": { "@oitmain/apollo-client-standalone": "^1.0.0", "apollo-link-ws": "^1.0.19", "subscriptions-transport-ws": "^0.9.16" } } This small node script takes a graphql subscription as its argument, and by just changing the callback function, you can quickly test your subscriptions. In the end it really helped me determining what was wrong (which in the end wasn’t in the FE or the BE, with an obscure HAProxy configuration).Getting started with Rust - part 22020-02-15T00:00:00+01:002020-02-15T00:00:00+01:00http://www.smartjava.org/content/getting-started-with-rust-part-2%20copy<p>In the <a href="https://www.smartjava.org/content/getting-started-with-rust-part-1/">previous article</a> I started experimenting with Rust and build a very simple extension to git, which helps me remember the most recent branches I’ve been working on, or did PRs for. For this
second article, I wanted to do look at how you can do network programming in Rust.</p>
<p>(If you just want to skip to the code: <a href="https://github.com/josdirksen/dns-proxy-rust">https://github.com/josdirksen/dns-proxy-rusts</a>)</p>
<p>For this, and the next article, since I’ve only done the basic so far, we’ll look at the steps needed to create a DNS proxy in Rust. This DNS proxy in the end will have the following features:</p>
<ul>
<li>Proxy all incoming DNS requests to a real DNS server, and respond back to the client with the response from the real server.</li>
<li>Keep metrics of how often a specific host was queried.</li>
<li>Allow the use of <code class="highlighter-rouge">/etc/hosts</code> with wildcards as a source for A-Record queries.</li>
<li>And probably some other stuff, which I’ll think of when implementing.</li>
</ul>
<p>One of reasons I want something simple like this is that at work we use wildcard certificates to connect specific domains to appications. So <code class="highlighter-rouge">ui-dev.smartjava.org</code>, would route to a specific instance, while <code class="highlighter-rouge">ui-test.smartjava.org</code> would route to another one. In some cases though, for instance when debugging, or testing k8s or kafka configs, it is useful to route these domains to localhost, or another host completely. We can do this by adding all the domains, including the subdomains to your <code class="highlighter-rouge">/etc/hosts</code> file, which solves part of the problem, but that only works when working directly on your local machine. If you start containers in docker, they won’t know about these changes. So with a simple DNS proxy, we can have docker instances use the information from
the docker host for routing.
And of course I’m just interested to see what kind of DNs queries my machine is making.</p>
<p>For this first article we’ll focus on the first of these points, and try to get the basic application up and running. As a more technical / Rust goal I wanted to mostly learn about:</p>
<ul>
<li>Network programming</li>
<li>async/await</li>
<li>Indirectly about the whole ownership model</li>
</ul>
<p>Another change I made in regards to the previous article, was that I switched editors a couple of times. Before going into the details on the code I’ve written, a quick note on that.</p>
<h2 id="visual-studio-code-as-rust-editor">Visual Studio Code as Rust Editor</h2>
<p>For my day job I do a lot of Kotlin, and you don’t really have much choice of editor. You just use Intellij Idea. Which is a great editor, but can sometimes become slow and unresponsive, since it’s got so many features, checks and other smart things running in the background, analyzing your code. In the previous article, I used the Intellij Rust plugin to write the git extension, but for this DNS example I’ve switched to using Visual Studio Code.</p>
<p><img src="../../images/rust-dns-2.png" alt="Visual studio code" /></p>
<p>Code has different plugins you can use for editing Rust, and I tried the following two:</p>
<ul>
<li><a href="https://marketplace.visualstudio.com/items?itemName=rust-lang.rust">The official RLS Extension</a>:</li>
<li><a href="https://marketplace.visualstudio.com/items?itemName=matklad.rust-analyzer">Rust analyzer Extension</a>:</li>
</ul>
<p>I started off with the offical extension, and initially it worked great. I got code completion, could quickly skip through source code, and had nice inline comments. It also provided integration with the Code tasks system to easily run tasks ons <code class="highlighter-rouge">cargo</code>. However, when I started adding the <code class="highlighter-rouge">async</code>/<code class="highlighter-rouge">await</code> stuff, it just stopped working. I still had some code highlighting, and it showed the errors (the squiggly lines), but there was no code completion anymore, and jumping to implementations also pretty much stopped working. Apparently this is a know issue, and is caused by the macros from Tokio, which I used to add async support to my code.
At that point I, momentarily, gave up on Rust in Code and switched back to IntelliJ. But, unfortunately, with the Tokio stuff added, IntelliJ was giving the same kind of errors. So back to Visual Studio Code, and then I tried the <em>Rust Analyzer</em> extension. This one was actually working quite ok with the async stuff, so I’ve written the rest of the product with that extension.</p>
<p>( There is also <a href="https://marketplace.visualstudio.com/items?itemName=kalitaalexey.vscode-rust">Another rust with RLS extension</a>, on the marketplace, but I didn’t test that one.)</p>
<p>I’ll check back in the near future how the various extensions evolve, but at least I’ve now got a light weight solution to work with in Visual Studio Code, which works quite nice with the tasks in Cargo.</p>
<h2 id="quick-introduction-on-udp">Quick introduction on UDP</h2>
<p>For this first version we’re just going to create a DNS Proxy which works with UDP. You can also run DNS queries over TCP, so we’ll add that in the next article. UDP is a connection-less protocol so we need to take that into account when creating our service. The following image from this slidedeck (<a href="https://slideplayer.com/slide/13399183/">Introduction into DNS</a>), provides a nice explanation of what we need to do:</p>
<p><img src="../../images/rust-dns-3.png" alt="UDP explanation" /></p>
<p>So we need to write the following:</p>
<ol>
<li>An UDP server which can receive datagrams from a DNS client (we’ll just use <code class="highlighter-rouge">dig</code> as the client)</li>
<li>When we receive an UDP request, we use an UDP client to forward the request to an external DNS server.</li>
<li>When that external server responds with a DNS answers, we forward it back to the initial UDP client (<code class="highlighter-rouge">dig</code>).</li>
</ol>
<p>First of, lets look at the libraries used for this proxy</p>
<h2 id="cargo-configuration">Cargo configuration</h2>
<p>We use the following <code class="highlighter-rouge">cargo.toml</code>.</p>
<div class="language-toml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nn">[package]</span>
<span class="py">name</span> <span class="p">=</span> <span class="s">"rust-dns"</span>
<span class="py">version</span> <span class="p">=</span> <span class="s">"0.1.0"</span>
<span class="py">authors</span> <span class="p">=</span> <span class="p">[</span><span class="s">"Jos Dirksen <jos.dirksen@gmail.com>"</span><span class="p">]</span>
<span class="py">edition</span> <span class="p">=</span> <span class="s">"2018"</span>
<span class="nn">[dependencies]</span>
<span class="py">dns-parser</span> <span class="p">=</span> <span class="s">"0.8.0"</span>
<span class="py">simplelog</span> <span class="p">=</span> <span class="s">"^0.7.4"</span>
<span class="py">log</span> <span class="p">=</span> <span class="s">"0.4.8"</span>
<span class="py">tokio</span> <span class="p">=</span> <span class="err">{</span> <span class="err">version</span> <span class="err">=</span> <span class="s">"0.2"</span><span class="p">,</span> <span class="err">features</span> <span class="err">=</span> <span class="nn">["full"]</span> <span class="err">}</span>
</code></pre></div></div>
<p>We’ve got the following libraries in here:</p>
<ul>
<li><strong>dns-parser</strong>: This library can parse incoming byte streams to an UDP packet. We don’t do much with this yet, just use it for some logging to see what is happening.</li>
<li><strong>simplelog</strong> and <strong>log</strong>: These two libraries provide logging functionality. The <strong>log</strong> library is an abstraction on top of the different logging libraries out there, and with <strong>simplelog</strong> we just get a simple logger, with colors.</li>
<li><strong>tokio</strong>: Rust has gotten async/await support a couple of months ago, and to use this you need a library that provides the execution runtime. Tokio is the best known one for this.</li>
</ul>
<p>Before I’ll show my simple code, a quick sidestep on the whole async/await stuff.</p>
<h2 id="asyncawait">Async/await</h2>
<p>With async/await we get a lot of syntactic sugar on how to work with futures. So we can write asynchronous code, which can be read as sequential code. Rust has a complete <a href="https://rust-lang.github.io/async-book/01_getting_started/04_async_await_primer.html">async handbook</a> (which I’ve only skimmed through so far) explaining this concept.</p>
<p>Basically what it allows is to write code like this:</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">async</span> <span class="k">fn</span> <span class="nf">function_1</span><span class="p">()</span> <span class="p">{</span> <span class="o">...</span> <span class="p">}</span>
<span class="n">async</span> <span class="k">fn</span> <span class="nf">function_2</span><span class="p">()</span> <span class="p">{</span> <span class="o">...</span> <span class="p">}</span>
<span class="nd">#[tokio</span><span class="err">::</span><span class="nd">main]</span>
<span class="n">async</span> <span class="k">fn</span> <span class="nf">main</span><span class="p">()</span> <span class="k">-></span> <span class="n">Result</span><span class="o"><</span><span class="p">(),</span> <span class="nb">Box</span><span class="o"><</span><span class="n">dyn</span> <span class="n">Error</span><span class="o">>></span> <span class="p">{</span>
<span class="nf">function_1</span><span class="p">()</span><span class="py">.await</span><span class="o">?</span>
<span class="nf">function_1</span><span class="p">()</span><span class="py">.await</span><span class="o">?</span>
<span class="nf">Ok</span><span class="p">(())</span>
<span class="p">}</span>
</code></pre></div></div>
<p>I hadn’t seen then question mark thingy <code class="highlighter-rouge">?</code> at the end of a function call beforehand, but it appears to be some syntactic sugar for dealing with <code class="highlighter-rouge">Result</code> types. It allows us to just chain together functions that return a <code class="highlighter-rouge">Result</code> without having to <code class="highlighter-rouge">and_then</code> or pattern match the results. Basically a minimal version of Haskell’s <code class="highlighter-rouge">do-notation</code> or Scala’s <code class="highlighter-rouge">for-comprehensions</code>.</p>
<p>Now let’s look at the code</p>
<h2 id="the-code">The code</h2>
<p>The code in itself isn’t that large, so I’ll first show the complete project so far with all its comments, and below that explain some of the stuff I was struggling with, or found interesting.</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">use</span> <span class="nn">dns_parser</span><span class="p">::</span><span class="nn">rdata</span><span class="p">::</span><span class="nn">a</span><span class="p">::</span><span class="n">Record</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">dns_parser</span><span class="p">::{</span><span class="n">Builder</span><span class="p">,</span> <span class="n">Error</span> <span class="k">as</span> <span class="n">DNSError</span><span class="p">,</span> <span class="n">Packet</span><span class="p">,</span> <span class="n">RData</span><span class="p">,</span> <span class="n">ResponseCode</span><span class="p">};</span>
<span class="k">use</span> <span class="nn">dns_parser</span><span class="p">::{</span><span class="n">QueryClass</span><span class="p">,</span> <span class="n">QueryType</span><span class="p">};</span>
<span class="k">use</span> <span class="nn">io</span><span class="p">::</span><span class="n">Result</span> <span class="k">as</span> <span class="n">ioResult</span><span class="p">;</span>
<span class="k">use</span> <span class="k">log</span><span class="p">::</span><span class="o">*</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">simplelog</span><span class="p">::{</span><span class="n">Config</span><span class="p">,</span> <span class="n">LevelFilter</span><span class="p">,</span> <span class="n">TermLogger</span><span class="p">,</span> <span class="n">TerminalMode</span><span class="p">};</span>
<span class="k">use</span> <span class="nn">std</span><span class="p">::</span><span class="nn">error</span><span class="p">::</span><span class="n">Error</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">std</span><span class="p">::</span><span class="nn">net</span><span class="p">::</span><span class="n">SocketAddr</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">std</span><span class="p">::</span><span class="nb">str</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">tokio</span><span class="p">::</span><span class="nn">net</span><span class="p">::</span><span class="n">UdpSocket</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">tokio</span><span class="p">::</span><span class="nn">prelude</span><span class="p">::</span><span class="o">*</span><span class="p">;</span>
<span class="c">///</span>
<span class="c">/// Simple dns proxy. For the first version, we'll just listen to</span>
<span class="c">/// an UDP socket, parse an incoming DNS query and write the output</span>
<span class="c">/// to the console.</span>
<span class="c">///</span>
<span class="c">/// Next steps:</span>
<span class="c">/// Support TCP protocol</span>
<span class="c">/// Keep track of hosts (and subhosts queried)</span>
<span class="c">/// Allow resolving through local host file</span>
<span class="c">/// Add error handling</span>
<span class="c">///</span>
<span class="nd">#[tokio</span><span class="err">::</span><span class="nd">main]</span>
<span class="n">async</span> <span class="k">fn</span> <span class="nf">main</span><span class="p">()</span> <span class="k">-></span> <span class="n">Result</span><span class="o"><</span><span class="p">(),</span> <span class="nb">Box</span><span class="o"><</span><span class="n">dyn</span> <span class="n">Error</span><span class="o">>></span> <span class="p">{</span>
<span class="nf">init_logging</span><span class="p">();</span>
<span class="nd">info!</span><span class="p">(</span><span class="s">"Starting server, setting up listener for port 12345"</span><span class="p">);</span>
<span class="c">// chain the await calls using '?'</span>
<span class="k">let</span> <span class="n">listen_socket</span> <span class="o">=</span> <span class="nf">create_udp_socket_receiver</span><span class="p">(</span><span class="s">"0.0.0.0:12345"</span><span class="p">)</span><span class="py">.await</span><span class="o">?</span><span class="p">;</span>
<span class="k">let</span> <span class="n">sender_socket</span> <span class="o">=</span> <span class="nf">create_udp_socket_sender</span><span class="p">()</span><span class="py">.await</span><span class="o">?</span><span class="p">;</span>
<span class="nf">start_listening_udp</span><span class="p">(</span><span class="n">listen_socket</span><span class="p">,</span> <span class="n">sender_socket</span><span class="p">)</span><span class="py">.await</span><span class="o">?</span><span class="p">;</span>
<span class="nf">Ok</span><span class="p">(())</span>
<span class="p">}</span>
<span class="c">///</span>
<span class="c">/// Create an async UDP socket.</span>
<span class="c">///</span>
<span class="n">async</span> <span class="k">fn</span> <span class="nf">create_udp_socket_receiver</span><span class="p">(</span><span class="n">host</span><span class="p">:</span> <span class="o">&</span><span class="nb">str</span><span class="p">)</span> <span class="k">-></span> <span class="n">ioResult</span><span class="o"><</span><span class="n">UdpSocket</span><span class="o">></span> <span class="p">{</span>
<span class="nd">debug!</span><span class="p">(</span><span class="s">"initializing listener udp socket on {}"</span><span class="p">,</span> <span class="n">host</span><span class="p">);</span>
<span class="k">let</span> <span class="n">socket</span> <span class="o">=</span> <span class="nn">UdpSocket</span><span class="p">::</span><span class="nf">bind</span><span class="p">(</span><span class="o">&</span><span class="n">host</span><span class="p">)</span><span class="py">.await</span><span class="o">?</span><span class="p">;</span>
<span class="k">return</span> <span class="nf">Ok</span><span class="p">(</span><span class="n">socket</span><span class="p">);</span>
<span class="p">}</span>
<span class="c">///</span>
<span class="c">/// Create the sender to forward the UDP request</span>
<span class="c">///</span>
<span class="n">async</span> <span class="k">fn</span> <span class="nf">create_udp_socket_sender</span><span class="p">()</span> <span class="k">-></span> <span class="n">ioResult</span><span class="o"><</span><span class="n">UdpSocket</span><span class="o">></span> <span class="p">{</span>
<span class="k">let</span> <span class="n">local_address</span> <span class="o">=</span> <span class="s">"0.0.0.0:0"</span><span class="p">;</span>
<span class="k">let</span> <span class="n">socket</span> <span class="o">=</span> <span class="nn">UdpSocket</span><span class="p">::</span><span class="nf">bind</span><span class="p">(</span><span class="n">local_address</span><span class="p">)</span><span class="py">.await</span><span class="o">?</span><span class="p">;</span>
<span class="k">let</span> <span class="n">socket_address</span><span class="p">:</span> <span class="n">SocketAddr</span> <span class="o">=</span> <span class="s">"8.8.8.8:53"</span>
<span class="py">.parse</span><span class="p">::</span><span class="o"><</span><span class="n">SocketAddr</span><span class="o">></span><span class="p">()</span>
<span class="nf">.expect</span><span class="p">(</span><span class="s">"Invalid forwarding address specified"</span><span class="p">);</span>
<span class="n">socket</span><span class="nf">.connect</span><span class="p">(</span><span class="o">&</span><span class="n">socket_address</span><span class="p">)</span><span class="py">.await</span><span class="o">?</span><span class="p">;</span>
<span class="nd">debug!</span><span class="p">(</span><span class="s">"initializing listener udp socket on {}"</span><span class="p">,</span> <span class="n">local_address</span><span class="p">);</span>
<span class="k">return</span> <span class="nf">Ok</span><span class="p">(</span><span class="n">socket</span><span class="p">);</span>
<span class="p">}</span>
<span class="c">///</span>
<span class="c">/// Asynchronously run the handling of incoming messages. Whenever something comes in on the socket_rcv_from</span>
<span class="c">/// we try and convert it to a DNS query, and log it to the output. UDP is a connectionless protocol, so we</span>
<span class="c">/// can use this socket to send and receive messages to our client and other servers.</span>
<span class="c">///</span>
<span class="n">async</span> <span class="k">fn</span> <span class="nf">start_listening_udp</span><span class="p">(</span>
<span class="k">mut</span> <span class="n">listen_socket</span><span class="p">:</span> <span class="n">UdpSocket</span><span class="p">,</span>
<span class="k">mut</span> <span class="n">sender_socket</span><span class="p">:</span> <span class="n">UdpSocket</span><span class="p">,</span>
<span class="p">)</span> <span class="k">-></span> <span class="n">ioResult</span><span class="o"><</span><span class="p">()</span><span class="o">></span> <span class="p">{</span>
<span class="c">// 1. Wait for a request from a DNS client.</span>
<span class="c">// 2. Then forward the request to a remote dns server</span>
<span class="c">// 3. The response from the remote DNS server is then send back to the initial client.</span>
<span class="k">loop</span> <span class="p">{</span>
<span class="k">let</span> <span class="p">(</span><span class="n">request</span><span class="p">,</span> <span class="n">peer</span><span class="p">)</span> <span class="o">=</span> <span class="nf">receive_request</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span> <span class="n">listen_socket</span><span class="p">)</span><span class="py">.await</span><span class="o">?</span><span class="p">;</span>
<span class="k">let</span> <span class="n">forward_response</span> <span class="o">=</span> <span class="nf">forward_request</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span> <span class="n">sender_socket</span><span class="p">,</span> <span class="o">&</span><span class="n">request</span><span class="p">[</span><span class="o">..</span><span class="p">])</span><span class="py">.await</span><span class="o">?</span><span class="p">;</span>
<span class="n">listen_socket</span><span class="nf">.send_to</span><span class="p">(</span><span class="o">&</span><span class="n">forward_response</span><span class="p">[</span><span class="o">..</span><span class="p">],</span> <span class="o">&</span><span class="n">peer</span><span class="p">)</span><span class="py">.await</span><span class="o">?</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="c">///</span>
<span class="c">/// Forward a request to the provided UDP socket, and wait for an answer.</span>
<span class="c">///</span>
<span class="n">async</span> <span class="k">fn</span> <span class="nf">forward_request</span><span class="p">(</span><span class="n">sender_socket</span><span class="p">:</span> <span class="o">&</span><span class="k">mut</span> <span class="n">UdpSocket</span><span class="p">,</span> <span class="n">request</span><span class="p">:</span> <span class="o">&</span><span class="p">[</span><span class="nb">u8</span><span class="p">])</span> <span class="k">-></span> <span class="n">ioResult</span><span class="o"><</span><span class="nb">Vec</span><span class="o"><</span><span class="nb">u8</span><span class="o">>></span> <span class="p">{</span>
<span class="k">let</span> <span class="k">mut</span> <span class="n">buf</span> <span class="o">=</span> <span class="p">[</span><span class="mi">0</span><span class="p">;</span> <span class="mi">4096</span><span class="p">];</span>
<span class="nd">info!</span><span class="p">(</span><span class="s">"Forwarding to target DNS"</span><span class="p">);</span>
<span class="n">sender_socket</span><span class="nf">.send</span><span class="p">(</span><span class="n">request</span><span class="p">)</span><span class="py">.await</span><span class="o">?</span><span class="p">;</span>
<span class="k">let</span> <span class="p">(</span><span class="n">amt</span><span class="p">,</span> <span class="n">_</span><span class="p">)</span> <span class="o">=</span> <span class="n">sender_socket</span><span class="nf">.recv_from</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span> <span class="n">buf</span><span class="p">)</span><span class="py">.await</span><span class="o">?</span><span class="p">;</span>
<span class="k">let</span> <span class="n">filled_buf</span> <span class="o">=</span> <span class="o">&</span><span class="k">mut</span> <span class="n">buf</span><span class="p">[</span><span class="o">..</span><span class="n">amt</span><span class="p">];</span>
<span class="c">// let answer_received = parse_incoming_stream(filled_buf).expect("Something went wrong");</span>
<span class="k">let</span> <span class="n">v</span> <span class="o">=</span> <span class="nn">Vec</span><span class="p">::</span><span class="nf">from</span><span class="p">(</span><span class="n">filled_buf</span><span class="p">);</span>
<span class="k">return</span> <span class="nf">Ok</span><span class="p">(</span><span class="n">v</span><span class="p">);</span>
<span class="p">}</span>
<span class="c">///</span>
<span class="c">/// Receive a request on the reference to the socket. We're not the owner, but we need a mutable</span>
<span class="c">/// reference. The result is a Vec, and the response address. Both are copies, which can be safely</span>
<span class="c">/// modified.</span>
<span class="c">///</span>
<span class="n">async</span> <span class="k">fn</span> <span class="nf">receive_request</span><span class="p">(</span><span class="n">from_socket</span><span class="p">:</span> <span class="o">&</span><span class="k">mut</span> <span class="n">UdpSocket</span><span class="p">)</span> <span class="k">-></span> <span class="n">ioResult</span><span class="o"><</span><span class="p">(</span><span class="nb">Vec</span><span class="o"><</span><span class="nb">u8</span><span class="o">></span><span class="p">,</span> <span class="n">SocketAddr</span><span class="p">)</span><span class="o">></span> <span class="p">{</span>
<span class="k">let</span> <span class="k">mut</span> <span class="n">buf</span> <span class="o">=</span> <span class="p">[</span><span class="mi">0</span><span class="p">;</span> <span class="mi">4096</span><span class="p">];</span>
<span class="k">let</span> <span class="p">(</span><span class="n">amt</span><span class="p">,</span> <span class="n">peer</span><span class="p">)</span> <span class="o">=</span> <span class="n">from_socket</span><span class="nf">.recv_from</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span> <span class="n">buf</span><span class="p">)</span><span class="py">.await</span><span class="o">?</span><span class="p">;</span>
<span class="k">let</span> <span class="n">filled_buf</span> <span class="o">=</span> <span class="o">&</span><span class="k">mut</span> <span class="n">buf</span><span class="p">[</span><span class="o">..</span><span class="n">amt</span><span class="p">];</span>
<span class="c">// info!("Received length {}", amt);</span>
<span class="c">// let packet_received = parse_incoming_stream(filled_buf).expect("Something went wrong");</span>
<span class="c">// info!("Received package {:?}", packet_received);</span>
<span class="k">let</span> <span class="n">v</span> <span class="o">=</span> <span class="nn">Vec</span><span class="p">::</span><span class="nf">from</span><span class="p">(</span><span class="n">filled_buf</span><span class="p">);</span>
<span class="k">return</span> <span class="nf">Ok</span><span class="p">((</span><span class="n">v</span><span class="p">,</span> <span class="n">peer</span><span class="p">));</span>
<span class="p">}</span>
<span class="c">///</span>
<span class="c">/// Parse the packet and return it</span>
<span class="c">///</span>
<span class="c">// fn parse_incoming_stream(incoming: &[u8]) -> Result<Packet, DNSError> {</span>
<span class="c">// let pkt = Packet::parse(incoming)?;</span>
<span class="c">// return Ok(pkt);</span>
<span class="c">// }</span>
<span class="c">///</span>
<span class="c">/// Initializes the logging library. We just simply log to console</span>
<span class="c">///</span>
<span class="k">fn</span> <span class="nf">init_logging</span><span class="p">()</span> <span class="p">{</span>
<span class="nn">TermLogger</span><span class="p">::</span><span class="nf">init</span><span class="p">(</span><span class="nn">LevelFilter</span><span class="p">::</span><span class="n">Debug</span><span class="p">,</span> <span class="nn">Config</span><span class="p">::</span><span class="nf">default</span><span class="p">(),</span> <span class="nn">TerminalMode</span><span class="p">::</span><span class="n">Mixed</span><span class="p">)</span><span class="nf">.unwrap</span><span class="p">();</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Let’s start with the <code class="highlighter-rouge">main</code> function. In our case this creates a client UDP socket, and a server UDP socket.</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="err">#</span><span class="p">[</span><span class="nn">tokio</span><span class="p">::</span><span class="n">main</span><span class="p">]</span>
<span class="n">async</span> <span class="k">fn</span> <span class="nf">main</span><span class="p">()</span> <span class="k">-></span> <span class="n">Result</span><span class="o"><</span><span class="p">(),</span> <span class="nb">Box</span><span class="o"><</span><span class="n">dyn</span> <span class="n">Error</span><span class="o">>></span> <span class="p">{</span>
<span class="nf">init_logging</span><span class="p">();</span>
<span class="nd">info!</span><span class="p">(</span><span class="s">"Starting server, setting up listener for port 12345"</span><span class="p">);</span>
<span class="k">let</span> <span class="n">listen_socket</span> <span class="o">=</span> <span class="nf">create_udp_socket_receiver</span><span class="p">(</span><span class="s">"0.0.0.0:12345"</span><span class="p">)</span><span class="py">.await</span><span class="o">?</span><span class="p">;</span>
<span class="k">let</span> <span class="n">sender_socket</span> <span class="o">=</span> <span class="nf">create_udp_socket_sender</span><span class="p">()</span><span class="py">.await</span><span class="o">?</span><span class="p">;</span>
<span class="nf">start_listening_udp</span><span class="p">(</span><span class="n">listen_socket</span><span class="p">,</span> <span class="n">sender_socket</span><span class="p">)</span><span class="py">.await</span><span class="o">?</span><span class="p">;</span>
<span class="nf">Ok</span><span class="p">(())</span>
<span class="p">}</span>
</code></pre></div></div>
<p>I’ll skip the implementations of the two <code class="highlighter-rouge">create</code> functions, which internally use the Tokio provided UDP network sockets. We use the Tokio provided ones, since they already work with async and await, so we don’t have to create and implement our own <code class="highlighter-rouge">Future</code> implementation on top of setting up the connections. When both of the sockets have been created, we pass them into the <code class="highlighter-rouge">start_listening_udp</code> function, which will continuously wait for new datagrams. Tokio provides an enhanced <code class="highlighter-rouge">main</code> function, which we can define as <code class="highlighter-rouge">async</code> itself, so we don’t have to do any blocking here.</p>
<p>In the function above you can also see how <code class="highlighter-rouge">await</code> together with the <code class="highlighter-rouge">?</code> function really allows for easily readable asynchronous code. The next interesting piece of code is the <code class="highlighter-rouge">start_listening_udp</code> function (in the future I want something similar for the TCP functionality):</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">async</span> <span class="k">fn</span> <span class="nf">start_listening_udp</span><span class="p">(</span>
<span class="k">mut</span> <span class="n">listen_socket</span><span class="p">:</span> <span class="n">UdpSocket</span><span class="p">,</span>
<span class="k">mut</span> <span class="n">sender_socket</span><span class="p">:</span> <span class="n">UdpSocket</span><span class="p">,</span>
<span class="p">)</span> <span class="k">-></span> <span class="n">ioResult</span><span class="o"><</span><span class="p">()</span><span class="o">></span> <span class="p">{</span>
<span class="k">loop</span> <span class="p">{</span>
<span class="k">let</span> <span class="p">(</span><span class="n">request</span><span class="p">,</span> <span class="n">peer</span><span class="p">)</span> <span class="o">=</span> <span class="nf">receive_request</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span> <span class="n">listen_socket</span><span class="p">)</span><span class="py">.await</span><span class="o">?</span><span class="p">;</span>
<span class="k">let</span> <span class="n">forward_response</span> <span class="o">=</span> <span class="nf">forward_request</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span> <span class="n">sender_socket</span><span class="p">,</span> <span class="o">&</span><span class="n">request</span><span class="p">[</span><span class="o">..</span><span class="p">])</span><span class="py">.await</span><span class="o">?</span><span class="p">;</span>
<span class="n">listen_socket</span><span class="nf">.send_to</span><span class="p">(</span><span class="o">&</span><span class="n">forward_response</span><span class="p">[</span><span class="o">..</span><span class="p">],</span> <span class="o">&</span><span class="n">peer</span><span class="p">)</span><span class="py">.await</span><span class="o">?</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>This function is a never-ending loop on top of three async functions. First we wait to receive a request on the socket our server is listening on. Once we get that request (e.g a message from <code class="highlighter-rouge">dig</code>), we call the <code class="highlighter-rouge">forward_request</code> function. This function will forward the request to a real DNS server, wait for a response, and return with the response. Finally we send the response back to our original client (<code class="highlighter-rouge">dig</code>) and wait for new requests to come in.
The nice thing about this code, is that it is very easy to reason about. The whole await/async stuff is nicely hidden from view, and we can sequence calls by usinig the question mark. I was impressed that a low-level language like Rust had such advanced constructs.</p>
<p>Let’s look at one of the functions, since the others are pretty much the same:</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">async</span> <span class="k">fn</span> <span class="nf">forward_request</span><span class="p">(</span><span class="n">sender_socket</span><span class="p">:</span> <span class="o">&</span><span class="k">mut</span> <span class="n">UdpSocket</span><span class="p">,</span> <span class="n">request</span><span class="p">:</span> <span class="o">&</span><span class="p">[</span><span class="nb">u8</span><span class="p">])</span> <span class="k">-></span> <span class="n">ioResult</span><span class="o"><</span><span class="nb">Vec</span><span class="o"><</span><span class="nb">u8</span><span class="o">>></span> <span class="p">{</span>
<span class="k">let</span> <span class="k">mut</span> <span class="n">buf</span> <span class="o">=</span> <span class="p">[</span><span class="mi">0</span><span class="p">;</span> <span class="mi">4096</span><span class="p">];</span>
<span class="nd">info!</span><span class="p">(</span><span class="s">"Forwarding to target DNS"</span><span class="p">);</span>
<span class="n">sender_socket</span><span class="nf">.send</span><span class="p">(</span><span class="n">request</span><span class="p">)</span><span class="py">.await</span><span class="o">?</span><span class="p">;</span>
<span class="k">let</span> <span class="p">(</span><span class="n">amt</span><span class="p">,</span> <span class="n">_</span><span class="p">)</span> <span class="o">=</span> <span class="n">sender_socket</span><span class="nf">.recv_from</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span> <span class="n">buf</span><span class="p">)</span><span class="py">.await</span><span class="o">?</span><span class="p">;</span>
<span class="k">let</span> <span class="n">filled_buf</span> <span class="o">=</span> <span class="o">&</span><span class="k">mut</span> <span class="n">buf</span><span class="p">[</span><span class="o">..</span><span class="n">amt</span><span class="p">];</span>
<span class="k">let</span> <span class="n">v</span> <span class="o">=</span> <span class="nn">Vec</span><span class="p">::</span><span class="nf">from</span><span class="p">(</span><span class="n">filled_buf</span><span class="p">);</span>
<span class="k">return</span> <span class="nf">Ok</span><span class="p">(</span><span class="n">v</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Here we have the same couple of <code class="highlighter-rouge">await?</code> steps. This time to make a call, and wait for a response. Easy right?</p>
<p>So finally when we run this, we get the following when we make a call:</p>
<p><img src="../../images/rust-dns-1.png" alt="Final output" /></p>
<p>Just like the previous project, it’s been really a fun process of doing this.</p>
<h2 id="conclusions">Conclusions</h2>
<p>So what was easy and what wasn’t?</p>
<ul>
<li>I’m still struggling with ownership, and when Rust is cleaning up variables used in functions. I’m slowly starting to understand it, but it does cause some strange and error messages when developing.</li>
<li>On those error messages. Whiile they usually are really helpful, sometimes it still takes a lot of time to actually solve the problem. This isn’t so much a Rust issue, as it is an issue for me that I’m still learning about specific concepts.</li>
<li>And it is always good to finish with a positivbe :) The whole async/await feels nice, works great and combined with the <code class="highlighter-rouge">?</code> thingy, results in compact and readable code.</li>
</ul>Jos DirksenIn the previous article I started experimenting with Rust and build a very simple extension to git, which helps me remember the most recent branches I’ve been working on, or did PRs for. For this second article, I wanted to do look at how you can do network programming in Rust. (If you just want to skip to the code: https://github.com/josdirksen/dns-proxy-rusts) For this, and the next article, since I’ve only done the basic so far, we’ll look at the steps needed to create a DNS proxy in Rust. This DNS proxy in the end will have the following features: Proxy all incoming DNS requests to a real DNS server, and respond back to the client with the response from the real server. Keep metrics of how often a specific host was queried. Allow the use of /etc/hosts with wildcards as a source for A-Record queries. And probably some other stuff, which I’ll think of when implementing. One of reasons I want something simple like this is that at work we use wildcard certificates to connect specific domains to appications. So ui-dev.smartjava.org, would route to a specific instance, while ui-test.smartjava.org would route to another one. In some cases though, for instance when debugging, or testing k8s or kafka configs, it is useful to route these domains to localhost, or another host completely. We can do this by adding all the domains, including the subdomains to your /etc/hosts file, which solves part of the problem, but that only works when working directly on your local machine. If you start containers in docker, they won’t know about these changes. So with a simple DNS proxy, we can have docker instances use the information from the docker host for routing. And of course I’m just interested to see what kind of DNs queries my machine is making. For this first article we’ll focus on the first of these points, and try to get the basic application up and running. As a more technical / Rust goal I wanted to mostly learn about: Network programming async/await Indirectly about the whole ownership model Another change I made in regards to the previous article, was that I switched editors a couple of times. Before going into the details on the code I’ve written, a quick note on that. Visual Studio Code as Rust Editor For my day job I do a lot of Kotlin, and you don’t really have much choice of editor. You just use Intellij Idea. Which is a great editor, but can sometimes become slow and unresponsive, since it’s got so many features, checks and other smart things running in the background, analyzing your code. In the previous article, I used the Intellij Rust plugin to write the git extension, but for this DNS example I’ve switched to using Visual Studio Code. Code has different plugins you can use for editing Rust, and I tried the following two: The official RLS Extension: Rust analyzer Extension: I started off with the offical extension, and initially it worked great. I got code completion, could quickly skip through source code, and had nice inline comments. It also provided integration with the Code tasks system to easily run tasks ons cargo. However, when I started adding the async/await stuff, it just stopped working. I still had some code highlighting, and it showed the errors (the squiggly lines), but there was no code completion anymore, and jumping to implementations also pretty much stopped working. Apparently this is a know issue, and is caused by the macros from Tokio, which I used to add async support to my code. At that point I, momentarily, gave up on Rust in Code and switched back to IntelliJ. But, unfortunately, with the Tokio stuff added, IntelliJ was giving the same kind of errors. So back to Visual Studio Code, and then I tried the Rust Analyzer extension. This one was actually working quite ok with the async stuff, so I’ve written the rest of the product with that extension. ( There is also Another rust with RLS extension, on the marketplace, but I didn’t test that one.) I’ll check back in the near future how the various extensions evolve, but at least I’ve now got a light weight solution to work with in Visual Studio Code, which works quite nice with the tasks in Cargo. Quick introduction on UDP For this first version we’re just going to create a DNS Proxy which works with UDP. You can also run DNS queries over TCP, so we’ll add that in the next article. UDP is a connection-less protocol so we need to take that into account when creating our service. The following image from this slidedeck (Introduction into DNS), provides a nice explanation of what we need to do: So we need to write the following: An UDP server which can receive datagrams from a DNS client (we’ll just use dig as the client) When we receive an UDP request, we use an UDP client to forward the request to an external DNS server. When that external server responds with a DNS answers, we forward it back to the initial UDP client (dig). First of, lets look at the libraries used for this proxy Cargo configuration We use the following cargo.toml. [package] name = "rust-dns" version = "0.1.0" authors = ["Jos Dirksen <jos.dirksen@gmail.com>"] edition = "2018" [dependencies] dns-parser = "0.8.0" simplelog = "^0.7.4" log = "0.4.8" tokio = { version = "0.2", features = ["full"] } We’ve got the following libraries in here: dns-parser: This library can parse incoming byte streams to an UDP packet. We don’t do much with this yet, just use it for some logging to see what is happening. simplelog and log: These two libraries provide logging functionality. The log library is an abstraction on top of the different logging libraries out there, and with simplelog we just get a simple logger, with colors. tokio: Rust has gotten async/await support a couple of months ago, and to use this you need a library that provides the execution runtime. Tokio is the best known one for this. Before I’ll show my simple code, a quick sidestep on the whole async/await stuff. Async/await With async/await we get a lot of syntactic sugar on how to work with futures. So we can write asynchronous code, which can be read as sequential code. Rust has a complete async handbook (which I’ve only skimmed through so far) explaining this concept. Basically what it allows is to write code like this: async fn function_1() { ... } async fn function_2() { ... } #[tokio::main] async fn main() -> Result<(), Box<dyn Error>> { function_1().await? function_1().await? Ok(()) } I hadn’t seen then question mark thingy ? at the end of a function call beforehand, but it appears to be some syntactic sugar for dealing with Result types. It allows us to just chain together functions that return a Result without having to and_then or pattern match the results. Basically a minimal version of Haskell’s do-notation or Scala’s for-comprehensions. Now let’s look at the code The code The code in itself isn’t that large, so I’ll first show the complete project so far with all its comments, and below that explain some of the stuff I was struggling with, or found interesting. use dns_parser::rdata::a::Record; use dns_parser::{Builder, Error as DNSError, Packet, RData, ResponseCode}; use dns_parser::{QueryClass, QueryType}; use io::Result as ioResult; use log::*; use simplelog::{Config, LevelFilter, TermLogger, TerminalMode}; use std::error::Error; use std::net::SocketAddr; use std::str; use tokio::net::UdpSocket; use tokio::prelude::*; /// /// Simple dns proxy. For the first version, we'll just listen to /// an UDP socket, parse an incoming DNS query and write the output /// to the console. /// /// Next steps: /// Support TCP protocol /// Keep track of hosts (and subhosts queried) /// Allow resolving through local host file /// Add error handling /// #[tokio::main] async fn main() -> Result<(), Box<dyn Error>> { init_logging(); info!("Starting server, setting up listener for port 12345"); // chain the await calls using '?' let listen_socket = create_udp_socket_receiver("0.0.0.0:12345").await?; let sender_socket = create_udp_socket_sender().await?; start_listening_udp(listen_socket, sender_socket).await?; Ok(()) } /// /// Create an async UDP socket. /// async fn create_udp_socket_receiver(host: &str) -> ioResult<UdpSocket> { debug!("initializing listener udp socket on {}", host); let socket = UdpSocket::bind(&host).await?; return Ok(socket); } /// /// Create the sender to forward the UDP request /// async fn create_udp_socket_sender() -> ioResult<UdpSocket> { let local_address = "0.0.0.0:0"; let socket = UdpSocket::bind(local_address).await?; let socket_address: SocketAddr = "8.8.8.8:53" .parse::<SocketAddr>() .expect("Invalid forwarding address specified"); socket.connect(&socket_address).await?; debug!("initializing listener udp socket on {}", local_address); return Ok(socket); } /// /// Asynchronously run the handling of incoming messages. Whenever something comes in on the socket_rcv_from /// we try and convert it to a DNS query, and log it to the output. UDP is a connectionless protocol, so we /// can use this socket to send and receive messages to our client and other servers. /// async fn start_listening_udp( mut listen_socket: UdpSocket, mut sender_socket: UdpSocket, ) -> ioResult<()> { // 1. Wait for a request from a DNS client. // 2. Then forward the request to a remote dns server // 3. The response from the remote DNS server is then send back to the initial client. loop { let (request, peer) = receive_request(&mut listen_socket).await?; let forward_response = forward_request(&mut sender_socket, &request[..]).await?; listen_socket.send_to(&forward_response[..], &peer).await?; } } /// /// Forward a request to the provided UDP socket, and wait for an answer. /// async fn forward_request(sender_socket: &mut UdpSocket, request: &[u8]) -> ioResult<Vec<u8>> { let mut buf = [0; 4096]; info!("Forwarding to target DNS"); sender_socket.send(request).await?; let (amt, _) = sender_socket.recv_from(&mut buf).await?; let filled_buf = &mut buf[..amt]; // let answer_received = parse_incoming_stream(filled_buf).expect("Something went wrong"); let v = Vec::from(filled_buf); return Ok(v); } /// /// Receive a request on the reference to the socket. We're not the owner, but we need a mutable /// reference. The result is a Vec, and the response address. Both are copies, which can be safely /// modified. /// async fn receive_request(from_socket: &mut UdpSocket) -> ioResult<(Vec<u8>, SocketAddr)> { let mut buf = [0; 4096]; let (amt, peer) = from_socket.recv_from(&mut buf).await?; let filled_buf = &mut buf[..amt]; // info!("Received length {}", amt); // let packet_received = parse_incoming_stream(filled_buf).expect("Something went wrong"); // info!("Received package {:?}", packet_received); let v = Vec::from(filled_buf); return Ok((v, peer)); } /// /// Parse the packet and return it /// // fn parse_incoming_stream(incoming: &[u8]) -> Result<Packet, DNSError> { // let pkt = Packet::parse(incoming)?; // return Ok(pkt); // } /// /// Initializes the logging library. We just simply log to console /// fn init_logging() { TermLogger::init(LevelFilter::Debug, Config::default(), TerminalMode::Mixed).unwrap(); } Let’s start with the main function. In our case this creates a client UDP socket, and a server UDP socket. #[tokio::main] async fn main() -> Result<(), Box<dyn Error>> { init_logging(); info!("Starting server, setting up listener for port 12345"); let listen_socket = create_udp_socket_receiver("0.0.0.0:12345").await?; let sender_socket = create_udp_socket_sender().await?; start_listening_udp(listen_socket, sender_socket).await?; Ok(()) } I’ll skip the implementations of the two create functions, which internally use the Tokio provided UDP network sockets. We use the Tokio provided ones, since they already work with async and await, so we don’t have to create and implement our own Future implementation on top of setting up the connections. When both of the sockets have been created, we pass them into the start_listening_udp function, which will continuously wait for new datagrams. Tokio provides an enhanced main function, which we can define as async itself, so we don’t have to do any blocking here. In the function above you can also see how await together with the ? function really allows for easily readable asynchronous code. The next interesting piece of code is the start_listening_udp function (in the future I want something similar for the TCP functionality): async fn start_listening_udp( mut listen_socket: UdpSocket, mut sender_socket: UdpSocket, ) -> ioResult<()> { loop { let (request, peer) = receive_request(&mut listen_socket).await?; let forward_response = forward_request(&mut sender_socket, &request[..]).await?; listen_socket.send_to(&forward_response[..], &peer).await?; } } This function is a never-ending loop on top of three async functions. First we wait to receive a request on the socket our server is listening on. Once we get that request (e.g a message from dig), we call the forward_request function. This function will forward the request to a real DNS server, wait for a response, and return with the response. Finally we send the response back to our original client (dig) and wait for new requests to come in. The nice thing about this code, is that it is very easy to reason about. The whole await/async stuff is nicely hidden from view, and we can sequence calls by usinig the question mark. I was impressed that a low-level language like Rust had such advanced constructs. Let’s look at one of the functions, since the others are pretty much the same: async fn forward_request(sender_socket: &mut UdpSocket, request: &[u8]) -> ioResult<Vec<u8>> { let mut buf = [0; 4096]; info!("Forwarding to target DNS"); sender_socket.send(request).await?; let (amt, _) = sender_socket.recv_from(&mut buf).await?; let filled_buf = &mut buf[..amt]; let v = Vec::from(filled_buf); return Ok(v); } Here we have the same couple of await? steps. This time to make a call, and wait for a response. Easy right? So finally when we run this, we get the following when we make a call: Just like the previous project, it’s been really a fun process of doing this. Conclusions So what was easy and what wasn’t? I’m still struggling with ownership, and when Rust is cleaning up variables used in functions. I’m slowly starting to understand it, but it does cause some strange and error messages when developing. On those error messages. Whiile they usually are really helpful, sometimes it still takes a lot of time to actually solve the problem. This isn’t so much a Rust issue, as it is an issue for me that I’m still learning about specific concepts. And it is always good to finish with a positivbe :) The whole async/await feels nice, works great and combined with the ? thingy, results in compact and readable code.Getting started with Rust - part 12020-02-10T00:00:00+01:002020-02-10T00:00:00+01:00http://www.smartjava.org/content/getting-started-with-rust-part-1<p>I decided to look a bit into Rust. I’ve been hearing great things about it, the community seems to be (mostly) very inviting and open, and the language in itself seems to have a nice mix of modern features (lambdas, boxed types, extensions), while still allowing for low-level systems programming.
The initial idea was to work through the <a href="https://doc.rust-lang.org/book/">The rust programming languagebook</a> and write a bit about that. But that quickly became very boring for me, so I just decided to start writing some small tools to see what the language can do and how to use the various features of the language. So for a first program, I wanted to write something useful.</p>
<p>At my current project we work with lots of branches, each feature/bug/ticket has its own branch and with reviews, PRs different projects I sometimes lose track of what I exactly named my branch I was working on. I could of course just check GitHub, or run some magical git command, but this would be a nice small tool to write with Rust. This simple tool should do this:</p>
<ul>
<li>Look at the current list of <code class="highlighter-rouge">git</code> branches in a specific directory.</li>
<li>Determine the hash and date of the last commit of that specific branch.</li>
<li>Print out the list of last commits per branch, ordered by date.</li>
</ul>
<p>That way I can quickly see on which branch I was working, and which branches were recently updated, without having to switch to each branch, and do <code class="highlighter-rouge">git log</code> and such.</p>
<p>Before we look at the code, a very big disclaimer. I haven’t looked at any optimizations yet, this was merely an exercise in Rust to get a working program. So any experienced Rust programmer will probably spot a whole lot of bad practices.</p>
<h2 id="the-final-result">The final result</h2>
<p>First, let’s see what the final program looks like:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ ./rust-git ~/dev/git/some/project
Using /Users/jos/dev/git/some/project as git repository to analyse.
2020-02-10 15:18:22 UTC : d3161dedcf7a99d3991b0492fb08285b245c93eb : dev####
2020-02-10 11:42:47 UTC : 0b75ba1d2f745ea6c0950646b92ec8480684472d : set##########################
2020-02-10 07:09:51 UTC : cdf7bbc0aa090d87ca8bcdebbe0bb1df105ca1c2 : cha#######################
2020-02-07 13:32:56 UTC : d52cdd456b2a1c2fd406a46601bad2d785a32f1e : rem########################
2020-02-06 12:36:11 UTC : edc540c8c79d9fff3e713bba4f6514d018a69df7 : sen################################
2020-02-05 13:57:05 UTC : fbdf684693a1791fecccc37d0c894fab619831fa : fix######################
2020-01-31 10:32:18 UTC : 8d35760cf0cc588b25d6566d4ef9771d172a23f4 : set###################
2020-01-27 10:22:35 UTC : 91ea1fabf9f7e6058a3e9f3e6b69fb87930786da : not#############
...
</code></pre></div></div>
<p>It actually uses color, so it looks much better when you actually run it. I’ve hashed the biggest part of the branches for this example, since I’m testing it on a work project. But you probably get the idea. It shows the branches, when was the last commit, and what is the hash of that commmit.</p>
<p><img src="../../images/rust-git-2.png" alt="With colors" /></p>
<p>Nothing too special. But fun to create. So without further ado, the code.</p>
<h2 id="the-code">The code</h2>
<p>We’ll start with the Toml. Using crates wasn’t so different as using any of the package managers like npm, Gradle, Maven, SBT and such. So for this sample I ended up with this:</p>
<div class="language-toml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nn">[package]</span>
<span class="py">name</span> <span class="p">=</span> <span class="s">"rust-git"</span>
<span class="py">version</span> <span class="p">=</span> <span class="s">"0.1.0"</span>
<span class="py">authors</span> <span class="p">=</span> <span class="p">[</span><span class="s">"Jos Dirksen <jos.dirksen@gmail.com>"</span><span class="p">]</span>
<span class="py">edition</span> <span class="p">=</span> <span class="s">"2018"</span>
<span class="nn">[dependencies]</span>
<span class="py">git2</span> <span class="p">=</span> <span class="s">"0.10"</span>
<span class="py">chrono</span> <span class="p">=</span> <span class="s">"0.4.10"</span>
<span class="py">libc</span> <span class="p">=</span> <span class="s">"0.2.66"</span>
<span class="py">colored</span> <span class="p">=</span> <span class="s">"1.9.2"</span>
</code></pre></div></div>
<p>Basically I used the dependencies for this:</p>
<ul>
<li><code class="highlighter-rouge">git2</code>: Provides access to git functionality by calling into <code class="highlighter-rouge">libgit2</code>.</li>
<li><code class="highlighter-rouge">chrono</code>: For dealing with and converting timestamp.</li>
<li><code class="highlighter-rouge">libc</code>: For an apparent issue with Rust, where you can’t pipe the output of a program to something like <code class="highlighter-rouge">head</code> or <code class="highlighter-rouge">less</code></li>
<li><code class="highlighter-rouge">colored</code>: Which provides extension functions (which I didn’t knew Rust has), to colorize output.</li>
</ul>
<p>This doesn’t look like much, but when doing typescript or react you often get a large number of transitive dependencies as well. So I ran into <code class="highlighter-rouge">cargo-tree</code>, where you can create tree of these dependencies:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>cargo-tree tree
rust-git v0.1.0 <span class="o">(</span>/Users/jos/dev/git/smartjava/articles/rust-git<span class="o">)</span>
├── chrono v0.4.10
│ ├── num-integer v0.1.42
│ │ └── num-traits v0.2.11
│ │ <span class="o">[</span>build-dependencies]
│ │ └── autocfg v1.0.0
│ │ <span class="o">[</span>build-dependencies]
│ │ └── autocfg v1.0.0 <span class="o">(</span><span class="k">*</span><span class="o">)</span>
│ ├── num-traits v0.2.11 <span class="o">(</span><span class="k">*</span><span class="o">)</span>
│ └── <span class="nb">time </span>v0.1.42
│ └── libc v0.2.66
├── colored v1.9.2
│ ├── atty v0.2.14
│ │ └── libc v0.2.66 <span class="o">(</span><span class="k">*</span><span class="o">)</span>
│ └── lazy_static v1.4.0
├── git2 v0.10.2
│ ├── bitflags v1.2.1
│ ├── libc v0.2.66 <span class="o">(</span><span class="k">*</span><span class="o">)</span>
│ ├── libgit2-sys v0.9.2
│ │ ├── libc v0.2.66 <span class="o">(</span><span class="k">*</span><span class="o">)</span>
│ │ ├── libssh2-sys v0.2.14
│ │ │ ├── libc v0.2.66 <span class="o">(</span><span class="k">*</span><span class="o">)</span>
│ │ │ ├── libz-sys v1.0.25
│ │ │ │ └── libc v0.2.66 <span class="o">(</span><span class="k">*</span><span class="o">)</span>
│ │ │ │ <span class="o">[</span>build-dependencies]
│ │ │ │ ├── cc v1.0.50
│ │ │ │ │ └── jobserver v0.1.21
│ │ │ │ │ └── libc v0.2.66 <span class="o">(</span><span class="k">*</span><span class="o">)</span>
│ │ │ │ └── pkg-config v0.3.17
│ │ │ └── openssl-sys v0.9.54
│ │ │ └── libc v0.2.66 <span class="o">(</span><span class="k">*</span><span class="o">)</span>
│ │ │ <span class="o">[</span>build-dependencies]
│ │ │ ├── autocfg v1.0.0 <span class="o">(</span><span class="k">*</span><span class="o">)</span>
│ │ │ ├── cc v1.0.50 <span class="o">(</span><span class="k">*</span><span class="o">)</span>
│ │ │ └── pkg-config v0.3.17 <span class="o">(</span><span class="k">*</span><span class="o">)</span>
│ │ │ <span class="o">[</span>build-dependencies]
│ │ │ ├── cc v1.0.50 <span class="o">(</span><span class="k">*</span><span class="o">)</span>
│ │ │ └── pkg-config v0.3.17 <span class="o">(</span><span class="k">*</span><span class="o">)</span>
│ │ ├── libz-sys v1.0.25 <span class="o">(</span><span class="k">*</span><span class="o">)</span>
│ │ └── openssl-sys v0.9.54 <span class="o">(</span><span class="k">*</span><span class="o">)</span>
│ │ <span class="o">[</span>build-dependencies]
│ │ ├── cc v1.0.50 <span class="o">(</span><span class="k">*</span><span class="o">)</span>
│ │ └── pkg-config v0.3.17 <span class="o">(</span><span class="k">*</span><span class="o">)</span>
│ ├── log v0.4.8
│ │ └── cfg-if v0.1.10
│ └── url v2.1.1
│ ├── idna v0.2.0
│ │ ├── matches v0.1.8
│ │ ├── unicode-bidi v0.3.4
│ │ │ └── matches v0.1.8 <span class="o">(</span><span class="k">*</span><span class="o">)</span>
│ │ └── unicode-normalization v0.1.12
│ │ └── smallvec v1.2.0
│ ├── matches v0.1.8 <span class="o">(</span><span class="k">*</span><span class="o">)</span>
│ └── percent-encoding v2.1.0
└── libc v0.2.66 <span class="o">(</span><span class="k">*</span><span class="o">)</span>
</code></pre></div></div>
<p>which isn’t that shocking. Lot’s of <code class="highlighter-rouge">build-dependencies</code> and a reasonable amount of other crates. Anyway, <code class="highlighter-rouge">cargo</code> so far just worked, which was nice, and it feels really quick. But, of course, this is really just a very simple project, so that’s probably to be expected.</p>
<p>Now the main code… and like I already mentioned. This is my first step into Rust, so bare with me.</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">use</span> <span class="nn">git2</span><span class="p">::{</span><span class="n">Repository</span><span class="p">,</span> <span class="n">BranchType</span><span class="p">,</span> <span class="n">Branch</span><span class="p">};</span>
<span class="k">use</span> <span class="nn">chrono</span><span class="p">::{</span><span class="n">Utc</span><span class="p">,</span> <span class="n">TimeZone</span><span class="p">,</span> <span class="n">DateTime</span><span class="p">};</span>
<span class="k">use</span> <span class="nn">std</span><span class="p">::</span><span class="nn">option</span><span class="p">::</span><span class="nb">Option</span> <span class="k">as</span> <span class="nb">Option</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">colored</span><span class="p">::</span><span class="o">*</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">std</span><span class="p">::</span><span class="n">env</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">std</span><span class="p">::</span><span class="nn">process</span><span class="p">::</span><span class="n">exit</span><span class="p">;</span>
<span class="k">struct</span> <span class="n">BranchCommitTime</span> <span class="p">{</span>
<span class="n">branch_name</span><span class="p">:</span> <span class="nb">String</span><span class="p">,</span>
<span class="n">last_commit</span><span class="p">:</span> <span class="n">DateTime</span><span class="o"><</span><span class="n">Utc</span><span class="o">></span><span class="p">,</span>
<span class="n">hash</span><span class="p">:</span> <span class="nb">String</span>
<span class="p">}</span>
<span class="k">fn</span> <span class="nf">main</span><span class="p">()</span> <span class="p">{</span>
<span class="c">// to allow piping the result to other programs</span>
<span class="c">//https://github.com/rust-lang/rust/issues/46016</span>
<span class="k">unsafe</span> <span class="p">{</span>
<span class="nn">libc</span><span class="p">::</span><span class="nf">signal</span><span class="p">(</span><span class="nn">libc</span><span class="p">::</span><span class="n">SIGPIPE</span><span class="p">,</span> <span class="nn">libc</span><span class="p">::</span><span class="n">SIG_DFL</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">let</span> <span class="n">args</span><span class="p">:</span> <span class="nb">Vec</span><span class="o"><</span><span class="nb">String</span><span class="o">></span> <span class="o">=</span> <span class="nn">env</span><span class="p">::</span><span class="nf">args</span><span class="p">()</span><span class="nf">.collect</span><span class="p">();</span>
<span class="k">match</span> <span class="n">args</span><span class="nf">.len</span><span class="p">()</span> <span class="p">{</span>
<span class="mi">1</span> <span class="k">=></span> <span class="p">{</span>
<span class="nd">println!</span><span class="p">(</span><span class="s">"{}"</span><span class="p">,</span><span class="s">"Using current dir as git repository to analyse."</span><span class="nf">.green</span><span class="p">());</span>
<span class="nf">show_branches_and_time</span><span class="p">(</span><span class="nn">env</span><span class="p">::</span><span class="nf">current_dir</span><span class="p">()</span><span class="nf">.unwrap</span><span class="p">()</span><span class="nf">.to_str</span><span class="p">()</span><span class="nf">.unwrap</span><span class="p">());</span>
<span class="p">},</span>
<span class="mi">2</span> <span class="k">=></span> <span class="p">{</span>
<span class="nd">println!</span><span class="p">(</span><span class="s">"{}{}{}"</span><span class="p">,</span> <span class="s">"Using "</span><span class="nf">.green</span><span class="p">()</span> <span class="p">,</span><span class="n">args</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span><span class="nf">.green</span><span class="p">(),</span> <span class="s">" as git repository to analyse."</span><span class="nf">.green</span><span class="p">());</span>
<span class="nf">show_branches_and_time</span><span class="p">(</span><span class="n">args</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span><span class="nf">.as_str</span><span class="p">());</span>
<span class="p">},</span>
<span class="n">_</span> <span class="k">=></span> <span class="p">{</span>
<span class="nd">println!</span><span class="p">(</span><span class="s">"{}"</span><span class="p">,</span> <span class="s">"Unexpected number of arguments"</span><span class="nf">.red</span><span class="p">());</span>
<span class="nf">exit</span><span class="p">(</span><span class="mi">1</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">};</span>
<span class="p">}</span>
<span class="k">fn</span> <span class="nf">show_branches_and_time</span><span class="p">(</span><span class="n">dir</span><span class="p">:</span> <span class="o">&</span><span class="nb">str</span><span class="p">)</span> <span class="p">{</span>
<span class="k">let</span> <span class="n">repo</span> <span class="o">=</span> <span class="k">match</span> <span class="nn">Repository</span><span class="p">::</span><span class="nf">open</span><span class="p">(</span><span class="n">dir</span><span class="p">)</span> <span class="p">{</span>
<span class="nf">Ok</span><span class="p">(</span><span class="n">repo</span><span class="p">)</span> <span class="k">=></span> <span class="n">repo</span><span class="p">,</span>
<span class="nf">Err</span><span class="p">(</span><span class="n">e</span><span class="p">)</span> <span class="k">=></span> <span class="nd">panic!</span><span class="p">(</span><span class="s">"failed to init: {}"</span><span class="p">,</span> <span class="n">e</span><span class="p">),</span>
<span class="p">};</span>
<span class="c">// get all the local branches, and for each get the name and last commit time</span>
<span class="c">// and return them as a new list of BranchCommitTime structs</span>
<span class="k">let</span> <span class="n">branches_list</span> <span class="o">=</span> <span class="nf">get_all_local_branches</span><span class="p">(</span><span class="o">&</span><span class="n">repo</span><span class="p">);</span>
<span class="k">let</span> <span class="n">bct_list</span> <span class="o">=</span> <span class="n">branches_list</span><span class="nf">.filter_map</span><span class="p">(|</span><span class="n">branch</span><span class="p">|</span> <span class="p">{</span>
<span class="c">// first get the name, or ignore the field when the name can't be determined</span>
<span class="k">return</span> <span class="k">match</span> <span class="n">branch</span> <span class="p">{</span>
<span class="nf">Ok</span><span class="p">((</span><span class="n">branch</span><span class="p">,</span> <span class="n">_</span><span class="p">))</span> <span class="k">=></span> <span class="p">{</span>
<span class="k">let</span> <span class="n">branch_name</span> <span class="o">=</span> <span class="nf">get_branch_name</span><span class="p">(</span><span class="o">&</span><span class="n">branch</span><span class="p">);</span>
<span class="k">let</span> <span class="n">last_commit</span> <span class="o">=</span> <span class="nf">get_branch_last_commit</span><span class="p">(</span><span class="o">&</span><span class="n">branch</span><span class="p">,</span> <span class="o">&</span><span class="n">repo</span><span class="p">);</span>
<span class="c">// flatmap these options to create an Option<BranchCommitTime></span>
<span class="k">let</span> <span class="n">bct</span> <span class="o">=</span> <span class="n">branch_name</span>
<span class="nf">.and_then</span><span class="p">(|</span><span class="n">n</span><span class="p">|</span> <span class="n">last_commit</span><span class="nf">.map</span><span class="p">(|</span><span class="n">t</span><span class="p">|</span>
<span class="n">BranchCommitTime</span> <span class="p">{</span> <span class="n">branch_name</span><span class="p">:</span> <span class="n">n</span><span class="p">,</span> <span class="n">last_commit</span><span class="p">:</span> <span class="n">t</span><span class="err">.</span><span class="mi">1</span><span class="p">,</span> <span class="n">hash</span><span class="p">:</span> <span class="n">t</span><span class="err">.</span><span class="mi">0</span> <span class="p">}</span>
<span class="p">));</span>
<span class="n">bct</span>
<span class="p">}</span>
<span class="nf">Err</span><span class="p">(</span><span class="n">_</span><span class="p">)</span> <span class="k">=></span> <span class="nb">None</span>
<span class="p">};</span>
<span class="p">});</span>
<span class="c">// collect the iterator in a vector so we can sort it. This has to</span>
<span class="c">// be a mutable one, since we do sorting in place. Finally print out</span>
<span class="c">// the sorted list to console.</span>
<span class="k">let</span> <span class="k">mut</span> <span class="n">bct_v</span><span class="p">:</span> <span class="nb">Vec</span><span class="o"><</span><span class="n">_</span><span class="o">></span> <span class="o">=</span> <span class="n">bct_list</span><span class="nf">.collect</span><span class="p">();</span>
<span class="n">bct_v</span><span class="nf">.sort_by</span><span class="p">(|</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">|</span> <span class="n">b</span><span class="py">.last_commit</span><span class="nf">.cmp</span><span class="p">(</span><span class="o">&</span><span class="n">a</span><span class="py">.last_commit</span><span class="p">));</span>
<span class="n">bct_v</span><span class="nf">.iter</span><span class="p">()</span><span class="nf">.for_each</span><span class="p">(|</span><span class="n">bc</span><span class="p">|</span> <span class="p">{</span>
<span class="c">// for article purposes we'll hash the names</span>
<span class="k">let</span> <span class="n">hashed_name</span> <span class="o">=</span> <span class="nf">hash_string</span><span class="p">(</span><span class="n">bc</span><span class="py">.branch_name</span><span class="nf">.as_str</span><span class="p">());</span>
<span class="nd">println!</span><span class="p">(</span><span class="s">"{} : {} : {}"</span><span class="p">,</span> <span class="n">bc</span><span class="py">.last_commit</span><span class="nf">.to_string</span><span class="p">()</span><span class="nf">.yellow</span><span class="p">(),</span> <span class="n">bc</span><span class="py">.hash</span><span class="p">,</span> <span class="n">hashed_name</span><span class="nf">.blue</span><span class="p">());</span>
<span class="p">}</span>
<span class="p">);</span>
<span class="p">}</span>
<span class="c">/// replace all the characters in a string, except the first 3</span>
<span class="k">fn</span> <span class="nf">hash_string</span><span class="p">(</span><span class="n">name</span><span class="p">:</span> <span class="o">&</span><span class="nb">str</span><span class="p">)</span> <span class="k">-></span> <span class="nb">String</span> <span class="p">{</span>
<span class="k">let</span> <span class="k">mut</span> <span class="n">hashed_name</span> <span class="o">=</span> <span class="nn">String</span><span class="p">::</span><span class="nf">from</span><span class="p">(</span><span class="s">""</span><span class="p">);</span>
<span class="k">for</span> <span class="p">(</span><span class="n">pos</span><span class="p">,</span> <span class="n">e</span><span class="p">)</span> <span class="n">in</span> <span class="n">name</span><span class="nf">.chars</span><span class="p">()</span><span class="nf">.enumerate</span><span class="p">()</span> <span class="p">{</span>
<span class="k">match</span> <span class="n">pos</span> <span class="p">{</span>
<span class="mi">0</span><span class="o">..=</span><span class="mi">2</span> <span class="k">=></span> <span class="n">hashed_name</span><span class="nf">.push</span><span class="p">(</span><span class="n">e</span><span class="p">),</span>
<span class="n">_</span> <span class="k">=></span> <span class="n">hashed_name</span><span class="nf">.push</span><span class="p">(</span><span class="sc">'#'</span><span class="p">)</span>
<span class="p">};</span>
<span class="p">}</span>
<span class="k">return</span> <span class="n">hashed_name</span>
<span class="p">}</span>
<span class="c">/// We get the name, and convert it to a string, so we don't</span>
<span class="c">/// run into ownership issues, or need to pass the lifetime around.</span>
<span class="k">fn</span> <span class="nf">get_branch_name</span><span class="p">(</span><span class="n">branch</span><span class="p">:</span> <span class="o">&</span><span class="n">Branch</span><span class="p">)</span> <span class="k">-></span> <span class="nb">Option</span><span class="o"><</span><span class="nb">String</span><span class="o">></span> <span class="p">{</span>
<span class="k">let</span> <span class="n">name</span> <span class="o">=</span> <span class="k">match</span> <span class="n">branch</span><span class="nf">.name</span><span class="p">()</span> <span class="p">{</span>
<span class="nf">Ok</span><span class="p">(</span><span class="n">r</span><span class="p">)</span> <span class="k">=></span> <span class="n">r</span><span class="nf">.map</span><span class="p">(|</span><span class="n">n</span><span class="p">|</span> <span class="nn">String</span><span class="p">::</span><span class="nf">from</span><span class="p">(</span><span class="n">n</span><span class="p">)),</span>
<span class="nf">Err</span><span class="p">(</span><span class="n">_</span><span class="p">)</span> <span class="k">=></span> <span class="nb">None</span>
<span class="p">};</span>
<span class="k">return</span> <span class="n">name</span><span class="p">;</span>
<span class="p">}</span>
<span class="c">/// get the last commit time from a branch, if it fails return none</span>
<span class="k">fn</span> <span class="nf">get_branch_last_commit</span><span class="p">(</span><span class="n">branch</span><span class="p">:</span> <span class="o">&</span><span class="n">Branch</span><span class="p">,</span> <span class="n">repo</span><span class="p">:</span> <span class="o">&</span><span class="n">Repository</span><span class="p">)</span> <span class="k">-></span> <span class="nb">Option</span><span class="o"><</span><span class="p">(</span><span class="nb">String</span><span class="p">,</span><span class="n">DateTime</span><span class="o"><</span><span class="n">Utc</span><span class="o">></span><span class="p">)</span><span class="o">></span> <span class="p">{</span>
<span class="k">let</span> <span class="n">p</span> <span class="o">=</span> <span class="n">branch</span><span class="nf">.get</span><span class="p">()</span><span class="nf">.target</span><span class="p">()</span><span class="nf">.and_then</span><span class="p">(|</span><span class="n">oid</span><span class="p">|</span> <span class="p">{</span>
<span class="k">let</span> <span class="n">commit</span> <span class="o">=</span> <span class="n">repo</span><span class="nf">.find_commit</span><span class="p">(</span><span class="n">oid</span><span class="p">);</span>
<span class="k">let</span> <span class="n">t</span> <span class="o">=</span> <span class="n">commit</span><span class="nf">.map</span><span class="p">(|</span><span class="n">c</span><span class="p">|</span>
<span class="p">(</span><span class="n">c</span><span class="nf">.id</span><span class="p">()</span><span class="nf">.to_string</span><span class="p">(),</span> <span class="n">Utc</span><span class="nf">.timestamp</span><span class="p">(</span><span class="n">c</span><span class="nf">.time</span><span class="p">()</span><span class="nf">.seconds</span><span class="p">(),</span> <span class="mi">0</span><span class="p">))</span>
<span class="p">);</span>
<span class="k">let</span> <span class="n">res</span> <span class="o">=</span> <span class="n">t</span><span class="nf">.ok</span><span class="p">();</span>
<span class="n">res</span>
<span class="p">});</span>
<span class="k">return</span> <span class="n">p</span><span class="p">;</span>
<span class="p">}</span>
<span class="c">/// Get all the local branches for the passed in repository</span>
<span class="k">fn</span> <span class="nf">get_all_local_branches</span><span class="p">(</span><span class="n">repo</span><span class="p">:</span> <span class="o">&</span><span class="n">Repository</span><span class="p">)</span> <span class="k">-></span> <span class="nn">git2</span><span class="p">::</span><span class="n">Branches</span> <span class="p">{</span>
<span class="k">let</span> <span class="n">branches_list</span> <span class="o">=</span> <span class="k">match</span> <span class="n">repo</span><span class="nf">.branches</span><span class="p">(</span><span class="nn">Option</span><span class="p">::</span><span class="nf">Some</span><span class="p">(</span><span class="nn">BranchType</span><span class="p">::</span><span class="n">Local</span><span class="p">))</span> <span class="p">{</span>
<span class="nf">Ok</span><span class="p">(</span><span class="n">br</span><span class="p">)</span> <span class="k">=></span> <span class="n">br</span><span class="p">,</span>
<span class="nf">Err</span><span class="p">(</span><span class="n">e</span><span class="p">)</span> <span class="k">=></span> <span class="nd">panic!</span><span class="p">(</span><span class="s">"failed to init: {}"</span><span class="p">,</span> <span class="n">e</span><span class="p">),</span>
<span class="p">};</span>
<span class="k">return</span> <span class="n">branches_list</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Nothing to special, but it allowed me to learn a couple of interesting concepts from Rust. The basic flow is shown in the code. For the end of this article, I’d like to listen a couple of small conclusions I made so far regarding Rust.</p>
<h2 id="the-conclusions-after-one-simple-rust-project">The conclusions after one simple Rust project</h2>
<p>Just a couple of points from my first project. In no particular order:</p>
<ul>
<li><strong>Rustc is very nice with exceptions and help how to solve it</strong>: I really like how the <code class="highlighter-rouge">rustc</code> compiler communicates errors. It not only tells you what is wrong, but also explains why it is, and what you might need to do to solve it. I don’t understand all the proposals yet, but it really helps with learning how the language, and its specific feature work (or should work).</li>
<li><strong>IntelliJ as editor has some rough edges</strong>: I’ve used the Intellij Rust plugin and that doesn’t feel really polished yet. I’ve been slowly moving away from IntelliJ for a couple of languages (Scala, Typescript and Javascript) to Visual Studio Code, which feels much leaner and quicker. I probably need to dive into the options available for Rust though.</li>
<li><strong>Option, Result feels natural</strong>: I really like that rust provides support for boxed types like Result and Option (and probably a lot more I haven’t looked at) out of the box. It really simplifies error handling, and feels much better than how it is done in Golang (at least for me).</li>
<li><strong>Missing <code class="highlighter-rouge">do</code> or <code class="highlighter-rouge">for-comprehension</code></strong>: But even though <code class="highlighter-rouge">Result</code> and <code class="highlighter-rouge">Option</code> are great, I do miss the <code class="highlighter-rouge">for-comprehensions</code> from Scala or Haskells <code class="highlighter-rouge">do</code> notation. After a quick search I already found a couple of libraries that add this syntactic sugar, so might look into that in the future as well. Kotlin, which I use daily at work, also doesn’t provide such a construct. But the <code class="highlighter-rouge">Arrow-kt</code> library provides a very nice extension for this to the core language.</li>
<li><strong>Pattern matching</strong>: I’ve mostly used pattern matching in Scala, where I really like it. In Kotlin the support isn’t that great, but in Rust so far it feels quite nice. I’m still struggling a bit with the syntax at certain points, but so far I’m quite liking it.</li>
<li><strong>Borrow, ownership, lifetimes</strong>: Apparently returning a <code class="highlighter-rouge">&str</code> is something complex. I’ve avoided the <code class="highlighter-rouge">Cow</code>, <code class="highlighter-rouge">Borrow</code>, <code class="highlighter-rouge">Into</code>, lifetime annotation stuff so far. After reading a bit, I see the need for it, and it looks like Rust has chosen a great approach. So in a future project I’ll dive into it in more depth.</li>
<li><strong>Odd lambda syntax</strong>: And finally, the lambda syntax takes some getting used to. It’s quite different from most languages I’ve used, but that’s probably just because all is new for me.</li>
</ul>
<p>All in all, I’m really pleasantly suprised by this language. It’s been really fun creating this simple tool, and I never really felt too much constrained by the language. While I struggled with some parts, the documentation and compiler hints (and a good amount of trial-and-error) got me where I wanted. Now just to think of something to build for the next experiment….</p>Jos DirksenI decided to look a bit into Rust. I’ve been hearing great things about it, the community seems to be (mostly) very inviting and open, and the language in itself seems to have a nice mix of modern features (lambdas, boxed types, extensions), while still allowing for low-level systems programming. The initial idea was to work through the The rust programming languagebook and write a bit about that. But that quickly became very boring for me, so I just decided to start writing some small tools to see what the language can do and how to use the various features of the language. So for a first program, I wanted to write something useful. At my current project we work with lots of branches, each feature/bug/ticket has its own branch and with reviews, PRs different projects I sometimes lose track of what I exactly named my branch I was working on. I could of course just check GitHub, or run some magical git command, but this would be a nice small tool to write with Rust. This simple tool should do this: Look at the current list of git branches in a specific directory. Determine the hash and date of the last commit of that specific branch. Print out the list of last commits per branch, ordered by date. That way I can quickly see on which branch I was working, and which branches were recently updated, without having to switch to each branch, and do git log and such. Before we look at the code, a very big disclaimer. I haven’t looked at any optimizations yet, this was merely an exercise in Rust to get a working program. So any experienced Rust programmer will probably spot a whole lot of bad practices. The final result First, let’s see what the final program looks like: $ ./rust-git ~/dev/git/some/project Using /Users/jos/dev/git/some/project as git repository to analyse. 2020-02-10 15:18:22 UTC : d3161dedcf7a99d3991b0492fb08285b245c93eb : dev#### 2020-02-10 11:42:47 UTC : 0b75ba1d2f745ea6c0950646b92ec8480684472d : set########################## 2020-02-10 07:09:51 UTC : cdf7bbc0aa090d87ca8bcdebbe0bb1df105ca1c2 : cha####################### 2020-02-07 13:32:56 UTC : d52cdd456b2a1c2fd406a46601bad2d785a32f1e : rem######################## 2020-02-06 12:36:11 UTC : edc540c8c79d9fff3e713bba4f6514d018a69df7 : sen################################ 2020-02-05 13:57:05 UTC : fbdf684693a1791fecccc37d0c894fab619831fa : fix###################### 2020-01-31 10:32:18 UTC : 8d35760cf0cc588b25d6566d4ef9771d172a23f4 : set################### 2020-01-27 10:22:35 UTC : 91ea1fabf9f7e6058a3e9f3e6b69fb87930786da : not############# ... It actually uses color, so it looks much better when you actually run it. I’ve hashed the biggest part of the branches for this example, since I’m testing it on a work project. But you probably get the idea. It shows the branches, when was the last commit, and what is the hash of that commmit. Nothing too special. But fun to create. So without further ado, the code. The code We’ll start with the Toml. Using crates wasn’t so different as using any of the package managers like npm, Gradle, Maven, SBT and such. So for this sample I ended up with this: [package] name = "rust-git" version = "0.1.0" authors = ["Jos Dirksen <jos.dirksen@gmail.com>"] edition = "2018" [dependencies] git2 = "0.10" chrono = "0.4.10" libc = "0.2.66" colored = "1.9.2" Basically I used the dependencies for this: git2: Provides access to git functionality by calling into libgit2. chrono: For dealing with and converting timestamp. libc: For an apparent issue with Rust, where you can’t pipe the output of a program to something like head or less colored: Which provides extension functions (which I didn’t knew Rust has), to colorize output. This doesn’t look like much, but when doing typescript or react you often get a large number of transitive dependencies as well. So I ran into cargo-tree, where you can create tree of these dependencies: $ cargo-tree tree rust-git v0.1.0 (/Users/jos/dev/git/smartjava/articles/rust-git) ├── chrono v0.4.10 │ ├── num-integer v0.1.42 │ │ └── num-traits v0.2.11 │ │ [build-dependencies] │ │ └── autocfg v1.0.0 │ │ [build-dependencies] │ │ └── autocfg v1.0.0 (*) │ ├── num-traits v0.2.11 (*) │ └── time v0.1.42 │ └── libc v0.2.66 ├── colored v1.9.2 │ ├── atty v0.2.14 │ │ └── libc v0.2.66 (*) │ └── lazy_static v1.4.0 ├── git2 v0.10.2 │ ├── bitflags v1.2.1 │ ├── libc v0.2.66 (*) │ ├── libgit2-sys v0.9.2 │ │ ├── libc v0.2.66 (*) │ │ ├── libssh2-sys v0.2.14 │ │ │ ├── libc v0.2.66 (*) │ │ │ ├── libz-sys v1.0.25 │ │ │ │ └── libc v0.2.66 (*) │ │ │ │ [build-dependencies] │ │ │ │ ├── cc v1.0.50 │ │ │ │ │ └── jobserver v0.1.21 │ │ │ │ │ └── libc v0.2.66 (*) │ │ │ │ └── pkg-config v0.3.17 │ │ │ └── openssl-sys v0.9.54 │ │ │ └── libc v0.2.66 (*) │ │ │ [build-dependencies] │ │ │ ├── autocfg v1.0.0 (*) │ │ │ ├── cc v1.0.50 (*) │ │ │ └── pkg-config v0.3.17 (*) │ │ │ [build-dependencies] │ │ │ ├── cc v1.0.50 (*) │ │ │ └── pkg-config v0.3.17 (*) │ │ ├── libz-sys v1.0.25 (*) │ │ └── openssl-sys v0.9.54 (*) │ │ [build-dependencies] │ │ ├── cc v1.0.50 (*) │ │ └── pkg-config v0.3.17 (*) │ ├── log v0.4.8 │ │ └── cfg-if v0.1.10 │ └── url v2.1.1 │ ├── idna v0.2.0 │ │ ├── matches v0.1.8 │ │ ├── unicode-bidi v0.3.4 │ │ │ └── matches v0.1.8 (*) │ │ └── unicode-normalization v0.1.12 │ │ └── smallvec v1.2.0 │ ├── matches v0.1.8 (*) │ └── percent-encoding v2.1.0 └── libc v0.2.66 (*) which isn’t that shocking. Lot’s of build-dependencies and a reasonable amount of other crates. Anyway, cargo so far just worked, which was nice, and it feels really quick. But, of course, this is really just a very simple project, so that’s probably to be expected. Now the main code… and like I already mentioned. This is my first step into Rust, so bare with me. use git2::{Repository, BranchType, Branch}; use chrono::{Utc, TimeZone, DateTime}; use std::option::Option as Option; use colored::*; use std::env; use std::process::exit; struct BranchCommitTime { branch_name: String, last_commit: DateTime<Utc>, hash: String } fn main() { // to allow piping the result to other programs //https://github.com/rust-lang/rust/issues/46016 unsafe { libc::signal(libc::SIGPIPE, libc::SIG_DFL); } let args: Vec<String> = env::args().collect(); match args.len() { 1 => { println!("{}","Using current dir as git repository to analyse.".green()); show_branches_and_time(env::current_dir().unwrap().to_str().unwrap()); }, 2 => { println!("{}{}{}", "Using ".green() ,args[1].green(), " as git repository to analyse.".green()); show_branches_and_time(args[1].as_str()); }, _ => { println!("{}", "Unexpected number of arguments".red()); exit(1); } }; } fn show_branches_and_time(dir: &str) { let repo = match Repository::open(dir) { Ok(repo) => repo, Err(e) => panic!("failed to init: {}", e), }; // get all the local branches, and for each get the name and last commit time // and return them as a new list of BranchCommitTime structs let branches_list = get_all_local_branches(&repo); let bct_list = branches_list.filter_map(|branch| { // first get the name, or ignore the field when the name can't be determined return match branch { Ok((branch, _)) => { let branch_name = get_branch_name(&branch); let last_commit = get_branch_last_commit(&branch, &repo); // flatmap these options to create an Option<BranchCommitTime> let bct = branch_name .and_then(|n| last_commit.map(|t| BranchCommitTime { branch_name: n, last_commit: t.1, hash: t.0 } )); bct } Err(_) => None }; }); // collect the iterator in a vector so we can sort it. This has to // be a mutable one, since we do sorting in place. Finally print out // the sorted list to console. let mut bct_v: Vec<_> = bct_list.collect(); bct_v.sort_by(|a, b| b.last_commit.cmp(&a.last_commit)); bct_v.iter().for_each(|bc| { // for article purposes we'll hash the names let hashed_name = hash_string(bc.branch_name.as_str()); println!("{} : {} : {}", bc.last_commit.to_string().yellow(), bc.hash, hashed_name.blue()); } ); } /// replace all the characters in a string, except the first 3 fn hash_string(name: &str) -> String { let mut hashed_name = String::from(""); for (pos, e) in name.chars().enumerate() { match pos { 0..=2 => hashed_name.push(e), _ => hashed_name.push('#') }; } return hashed_name } /// We get the name, and convert it to a string, so we don't /// run into ownership issues, or need to pass the lifetime around. fn get_branch_name(branch: &Branch) -> Option<String> { let name = match branch.name() { Ok(r) => r.map(|n| String::from(n)), Err(_) => None }; return name; } /// get the last commit time from a branch, if it fails return none fn get_branch_last_commit(branch: &Branch, repo: &Repository) -> Option<(String,DateTime<Utc>)> { let p = branch.get().target().and_then(|oid| { let commit = repo.find_commit(oid); let t = commit.map(|c| (c.id().to_string(), Utc.timestamp(c.time().seconds(), 0)) ); let res = t.ok(); res }); return p; } /// Get all the local branches for the passed in repository fn get_all_local_branches(repo: &Repository) -> git2::Branches { let branches_list = match repo.branches(Option::Some(BranchType::Local)) { Ok(br) => br, Err(e) => panic!("failed to init: {}", e), }; return branches_list; } Nothing to special, but it allowed me to learn a couple of interesting concepts from Rust. The basic flow is shown in the code. For the end of this article, I’d like to listen a couple of small conclusions I made so far regarding Rust. The conclusions after one simple Rust project Just a couple of points from my first project. In no particular order: Rustc is very nice with exceptions and help how to solve it: I really like how the rustc compiler communicates errors. It not only tells you what is wrong, but also explains why it is, and what you might need to do to solve it. I don’t understand all the proposals yet, but it really helps with learning how the language, and its specific feature work (or should work). IntelliJ as editor has some rough edges: I’ve used the Intellij Rust plugin and that doesn’t feel really polished yet. I’ve been slowly moving away from IntelliJ for a couple of languages (Scala, Typescript and Javascript) to Visual Studio Code, which feels much leaner and quicker. I probably need to dive into the options available for Rust though. Option, Result feels natural: I really like that rust provides support for boxed types like Result and Option (and probably a lot more I haven’t looked at) out of the box. It really simplifies error handling, and feels much better than how it is done in Golang (at least for me). Missing do or for-comprehension: But even though Result and Option are great, I do miss the for-comprehensions from Scala or Haskells do notation. After a quick search I already found a couple of libraries that add this syntactic sugar, so might look into that in the future as well. Kotlin, which I use daily at work, also doesn’t provide such a construct. But the Arrow-kt library provides a very nice extension for this to the core language. Pattern matching: I’ve mostly used pattern matching in Scala, where I really like it. In Kotlin the support isn’t that great, but in Rust so far it feels quite nice. I’m still struggling a bit with the syntax at certain points, but so far I’m quite liking it. Borrow, ownership, lifetimes: Apparently returning a &str is something complex. I’ve avoided the Cow, Borrow, Into, lifetime annotation stuff so far. After reading a bit, I see the need for it, and it looks like Rust has chosen a great approach. So in a future project I’ll dive into it in more depth. Odd lambda syntax: And finally, the lambda syntax takes some getting used to. It’s quite different from most languages I’ve used, but that’s probably just because all is new for me. All in all, I’m really pleasantly suprised by this language. It’s been really fun creating this simple tool, and I never really felt too much constrained by the language. While I struggled with some parts, the documentation and compiler hints (and a good amount of trial-and-error) got me where I wanted. Now just to think of something to build for the next experiment….Setting up a simple Kafka cluster with docker for testing2020-02-02T00:00:00+01:002020-02-02T00:00:00+01:00http://www.smartjava.org/content/setting-up-kafka-cluster-docker%20copy<p>In this short article we’ll have a quick look at how to set up a Kafka cluster locally, which can be easily
accessed from outside of the docker container. The reason for this article is that most of the example you can
find either provide a single Kafka instance, or provide a way to set up a Kafka cluster, whose hosts can only be accessed from within the docker container.</p>
<p>I ran into this issue when I needed to reproduce some strange issues with the Kafka instance provided by our private cloud provider. When maintenance happened and the nodes were being cycled, some topics and consumers seemed to completely loss track, and were unable to recover. They needed a restart of the service, before messages were being processed again.</p>
<h2 id="docker--kafka-setup">Docker / Kafka setup</h2>
<p>The main setup here is just a simple docker-compose file based on the great set of docker images from <a href="https://github.com/wurstmeister/kafka-docker">https://github.com/wurstmeister/kafka-docker</a>. So without much further introduction, we’ll look at the <code class="highlighter-rouge">docker-compose</code> file I used for this:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">version</span><span class="pi">:</span> <span class="s1">'</span><span class="s">2'</span>
<span class="na">services</span><span class="pi">:</span>
<span class="na">zookeeper</span><span class="pi">:</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">wurstmeister/zookeeper</span>
<span class="na">ports</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s2">"</span><span class="s">2181:2181"</span>
<span class="na">kafka-1</span><span class="pi">:</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">wurstmeister/kafka</span>
<span class="na">ports</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s2">"</span><span class="s">9095:9092"</span>
<span class="na">environment</span><span class="pi">:</span>
<span class="na">KAFKA_ADVERTISED_HOST_NAME</span><span class="pi">:</span> <span class="s">kafka1.test.local</span>
<span class="na">KAFKA_ADVERTISED_PORT</span><span class="pi">:</span> <span class="s">9095</span>
<span class="na">KAFKA_ZOOKEEPER_CONNECT</span><span class="pi">:</span> <span class="s">zookeeper:2181</span>
<span class="na">KAFKA_LOG_DIRS</span><span class="pi">:</span> <span class="s">/kafka/logs</span>
<span class="na">KAFKA_BROKER_ID</span><span class="pi">:</span> <span class="s">500</span>
<span class="na">KAFKA_offsets_topic_replication_factor</span><span class="pi">:</span> <span class="s">3</span>
<span class="na">volumes</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">/var/run/docker.sock:/var/run/docker.sock</span>
<span class="pi">-</span> <span class="s">${KAFKA_DATA}/500:/kafka</span>
<span class="na">kafka-2</span><span class="pi">:</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">wurstmeister/kafka</span>
<span class="na">ports</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s2">"</span><span class="s">9096:9092"</span>
<span class="na">environment</span><span class="pi">:</span>
<span class="na">KAFKA_ADVERTISED_HOST_NAME</span><span class="pi">:</span> <span class="s">kafka2.test.local</span>
<span class="na">KAFKA_ADVERTISED_PORT</span><span class="pi">:</span> <span class="s">9096</span>
<span class="na">KAFKA_ZOOKEEPER_CONNECT</span><span class="pi">:</span> <span class="s">zookeeper:2181</span>
<span class="na">KAFKA_LOG_DIRS</span><span class="pi">:</span> <span class="s">/kafka/logs</span>
<span class="na">KAFKA_BROKER_ID</span><span class="pi">:</span> <span class="s">501</span>
<span class="na">KAFKA_offsets_topic_replication_factor</span><span class="pi">:</span> <span class="s">3</span>
<span class="na">volumes</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">/var/run/docker.sock:/var/run/docker.sock</span>
<span class="pi">-</span> <span class="s">${KAFKA_DATA}/501:/kafka</span>
<span class="na">kafka-3</span><span class="pi">:</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">wurstmeister/kafka</span>
<span class="na">ports</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s2">"</span><span class="s">9097:9092"</span>
<span class="na">environment</span><span class="pi">:</span>
<span class="na">KAFKA_ADVERTISED_HOST_NAME</span><span class="pi">:</span> <span class="s">kafka1.test.local</span>
<span class="na">KAFKA_ADVERTISED_PORT</span><span class="pi">:</span> <span class="s">9097</span>
<span class="na">KAFKA_ZOOKEEPER_CONNECT</span><span class="pi">:</span> <span class="s">zookeeper:2181</span>
<span class="na">KAFKA_LOG_DIRS</span><span class="pi">:</span> <span class="s">/kafka/logs</span>
<span class="na">KAFKA_BROKER_ID</span><span class="pi">:</span> <span class="s">502</span>
<span class="na">KAFKA_offsets_topic_replication_factor</span><span class="pi">:</span> <span class="s">3</span>
<span class="na">volumes</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">/var/run/docker.sock:/var/run/docker.sock</span>
<span class="pi">-</span> <span class="s">${KAFKA_DATA}/502:/kafka</span>
</code></pre></div></div>
<p>What we see here is a simple <code class="highlighter-rouge">docker-compose</code> file where we define a single Zookeeper node and three kafka nodes. Note that
I’ve also expect the <code class="highlighter-rouge">KAFKA_DATA</code> variable to be set, which is used as an external volume. That way we don’t lose the data
when we remove the cluster. Let’s look a bit closer at the individual Kafka nodes:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="na">kafka-3</span><span class="pi">:</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">wurstmeister/kafka</span>
<span class="na">ports</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s2">"</span><span class="s">9097:9092"</span>
<span class="na">environment</span><span class="pi">:</span>
<span class="na">KAFKA_ADVERTISED_HOST_NAME</span><span class="pi">:</span> <span class="s">kafka1.test.local</span>
<span class="na">KAFKA_ADVERTISED_PORT</span><span class="pi">:</span> <span class="s">9097</span>
<span class="na">KAFKA_ZOOKEEPER_CONNECT</span><span class="pi">:</span> <span class="s">zookeeper:2181</span>
<span class="na">KAFKA_LOG_DIRS</span><span class="pi">:</span> <span class="s">/kafka/logs</span>
<span class="na">KAFKA_BROKER_ID</span><span class="pi">:</span> <span class="s">502</span>
<span class="na">KAFKA_offsets_topic_replication_factor</span><span class="pi">:</span> <span class="s">3</span>
<span class="na">volumes</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">/var/run/docker.sock:/var/run/docker.sock</span>
<span class="pi">-</span> <span class="s">${KAFKA_DATA}/502:/kafka</span>
</code></pre></div></div>
<p>Here we see the following:</p>
<ul>
<li>We expose the Kafka port <code class="highlighter-rouge">9092</code> on the external host on a unique port <code class="highlighter-rouge">9097</code> (we do this for each Kafka node in the cluster).</li>
<li>To make this work correctly we also set the <code class="highlighter-rouge">KAFKA_ADVERTISED_PORT</code> to <code class="highlighter-rouge">9097</code>, so clients can connect to the nodes correctly after discovery.</li>
<li>We need a unique host name for each node, if not. The cluster will complain the it already has a node with that name. So we set the unique
name using the <code class="highlighter-rouge">KAFKA_ADVERTISED_HOST_NAME</code>. We use <code class="highlighter-rouge">kafka1.test.local</code>, <code class="highlighter-rouge">kafka2.test.local</code> and <code class="highlighter-rouge">kafka3.test.local</code> as host name for our cluster.</li>
<li>Besides the settings above, we also give each node a unique id, using the <code class="highlighter-rouge">KAFKA_BROKER_ID</code> property.</li>
</ul>
<p>The final step to take to get this working is having to make sure we can resolve the hosts (<code class="highlighter-rouge">kafka1.test.local</code>) specified here correctly. We can run our own dns server for this, but it is easier to just update the local host file <code class="highlighter-rouge">/etc/hosts</code>.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>10.82.6.17 kafka1.test.local
10.82.6.17 kafka2.test.local
10.82.6.17 kafka3.test.local
</code></pre></div></div>
<h2 id="running-the-cluster">Running the cluster</h2>
<p>At this point we can simply start the cluster using <code class="highlighter-rouge">docker-compose</code>:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">export </span><span class="nv">KAFKA_DATA</span><span class="o">=</span>/Users/jos/dev/data/cer/kafka
<span class="nv">$ </span>docker-compose <span class="nt">-f</span> ./docker-compose-local-kafka-cluster.yml up <span class="nt">-d</span>
Starting local_zookeeper_1 ... <span class="k">done
</span>Creating local_kafka-3_1 ... <span class="k">done
</span>Creating local_kafka-1_1 ... <span class="k">done
</span>Creating local_kafka-2_1 ... <span class="k">done</span>
<span class="nv">$ </span>docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
da9d108cbc0e wurstmeister/kafka <span class="s2">"start-kafka.sh"</span> 7 seconds ago Up 6 seconds 0.0.0.0:9095->9092/tcp local_kafka-1_1
3819d7ca3d7c wurstmeister/kafka <span class="s2">"start-kafka.sh"</span> 7 seconds ago Up 6 seconds 0.0.0.0:9096->9092/tcp local_kafka-2_1
f8dc6ff937c6 wurstmeister/kafka <span class="s2">"start-kafka.sh"</span> 7 seconds ago Up 6 seconds 0.0.0.0:9097->9092/tcp local_kafka-3_1
62dfc7ea32c6 wurstmeister/zookeeper <span class="s2">"/bin/sh -c '/usr/sb…"</span> 2 days ago Up 6 seconds 22/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp local_zookeeper_1
</code></pre></div></div>
<p>We can quickly check which nodes are part of the cluster by running a command against zookeeper:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>docker <span class="nb">exec</span> <span class="nt">-ti</span> 62dfc7ea32c6 ./bin/zkCli.sh <span class="nb">ls</span> /brokers/ids
...
WATCHER::
WatchedEvent state:SyncConnected <span class="nb">type</span>:None path:null
<span class="o">[</span>501, 502, 500]
</code></pre></div></div>
<p>And that’s it. We’ve now got a cluster up and running, which we can use from outside the docker container, by just connecting to one of the three hosts. From the outside world, it’ll look like a valid cluster and we can test failover scenarios or other settings by simply bringing nodes down and seeing how the clients react.</p>
<h2 id="final-note">Final note</h2>
<p>On a final note, you might notice the <code class="highlighter-rouge">KAFKA_offsets_topic_replication_factor: 3</code> setting. This setting defines the replication factor of the topic used to store the consumers offset. In the default case this is set to 1. So the consumer offsets for a particular topic will only be present on a single node. If that node goes down, consumers will lose track of where they are, since they can’t update the consumer offsets. This was the main problem in our case. By setting this to <code class="highlighter-rouge">3</code> we could safely cycle the nodes of the cluster, without it affecting the coonsumers.</p>Jos DirksenIn this short article we’ll have a quick look at how to set up a Kafka cluster locally, which can be easily accessed from outside of the docker container. The reason for this article is that most of the example you can find either provide a single Kafka instance, or provide a way to set up a Kafka cluster, whose hosts can only be accessed from within the docker container. I ran into this issue when I needed to reproduce some strange issues with the Kafka instance provided by our private cloud provider. When maintenance happened and the nodes were being cycled, some topics and consumers seemed to completely loss track, and were unable to recover. They needed a restart of the service, before messages were being processed again. Docker / Kafka setup The main setup here is just a simple docker-compose file based on the great set of docker images from https://github.com/wurstmeister/kafka-docker. So without much further introduction, we’ll look at the docker-compose file I used for this: version: '2' services: zookeeper: image: wurstmeister/zookeeper ports: - "2181:2181" kafka-1: image: wurstmeister/kafka ports: - "9095:9092" environment: KAFKA_ADVERTISED_HOST_NAME: kafka1.test.local KAFKA_ADVERTISED_PORT: 9095 KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 KAFKA_LOG_DIRS: /kafka/logs KAFKA_BROKER_ID: 500 KAFKA_offsets_topic_replication_factor: 3 volumes: - /var/run/docker.sock:/var/run/docker.sock - ${KAFKA_DATA}/500:/kafka kafka-2: image: wurstmeister/kafka ports: - "9096:9092" environment: KAFKA_ADVERTISED_HOST_NAME: kafka2.test.local KAFKA_ADVERTISED_PORT: 9096 KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 KAFKA_LOG_DIRS: /kafka/logs KAFKA_BROKER_ID: 501 KAFKA_offsets_topic_replication_factor: 3 volumes: - /var/run/docker.sock:/var/run/docker.sock - ${KAFKA_DATA}/501:/kafka kafka-3: image: wurstmeister/kafka ports: - "9097:9092" environment: KAFKA_ADVERTISED_HOST_NAME: kafka1.test.local KAFKA_ADVERTISED_PORT: 9097 KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 KAFKA_LOG_DIRS: /kafka/logs KAFKA_BROKER_ID: 502 KAFKA_offsets_topic_replication_factor: 3 volumes: - /var/run/docker.sock:/var/run/docker.sock - ${KAFKA_DATA}/502:/kafka What we see here is a simple docker-compose file where we define a single Zookeeper node and three kafka nodes. Note that I’ve also expect the KAFKA_DATA variable to be set, which is used as an external volume. That way we don’t lose the data when we remove the cluster. Let’s look a bit closer at the individual Kafka nodes: kafka-3: image: wurstmeister/kafka ports: - "9097:9092" environment: KAFKA_ADVERTISED_HOST_NAME: kafka1.test.local KAFKA_ADVERTISED_PORT: 9097 KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 KAFKA_LOG_DIRS: /kafka/logs KAFKA_BROKER_ID: 502 KAFKA_offsets_topic_replication_factor: 3 volumes: - /var/run/docker.sock:/var/run/docker.sock - ${KAFKA_DATA}/502:/kafka Here we see the following: We expose the Kafka port 9092 on the external host on a unique port 9097 (we do this for each Kafka node in the cluster). To make this work correctly we also set the KAFKA_ADVERTISED_PORT to 9097, so clients can connect to the nodes correctly after discovery. We need a unique host name for each node, if not. The cluster will complain the it already has a node with that name. So we set the unique name using the KAFKA_ADVERTISED_HOST_NAME. We use kafka1.test.local, kafka2.test.local and kafka3.test.local as host name for our cluster. Besides the settings above, we also give each node a unique id, using the KAFKA_BROKER_ID property. The final step to take to get this working is having to make sure we can resolve the hosts (kafka1.test.local) specified here correctly. We can run our own dns server for this, but it is easier to just update the local host file /etc/hosts. 10.82.6.17 kafka1.test.local 10.82.6.17 kafka2.test.local 10.82.6.17 kafka3.test.local Running the cluster At this point we can simply start the cluster using docker-compose: $ export KAFKA_DATA=/Users/jos/dev/data/cer/kafka $ docker-compose -f ./docker-compose-local-kafka-cluster.yml up -d Starting local_zookeeper_1 ... done Creating local_kafka-3_1 ... done Creating local_kafka-1_1 ... done Creating local_kafka-2_1 ... done $ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES da9d108cbc0e wurstmeister/kafka "start-kafka.sh" 7 seconds ago Up 6 seconds 0.0.0.0:9095->9092/tcp local_kafka-1_1 3819d7ca3d7c wurstmeister/kafka "start-kafka.sh" 7 seconds ago Up 6 seconds 0.0.0.0:9096->9092/tcp local_kafka-2_1 f8dc6ff937c6 wurstmeister/kafka "start-kafka.sh" 7 seconds ago Up 6 seconds 0.0.0.0:9097->9092/tcp local_kafka-3_1 62dfc7ea32c6 wurstmeister/zookeeper "/bin/sh -c '/usr/sb…" 2 days ago Up 6 seconds 22/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp local_zookeeper_1 We can quickly check which nodes are part of the cluster by running a command against zookeeper: $ docker exec -ti 62dfc7ea32c6 ./bin/zkCli.sh ls /brokers/ids ... WATCHER:: WatchedEvent state:SyncConnected type:None path:null [501, 502, 500] And that’s it. We’ve now got a cluster up and running, which we can use from outside the docker container, by just connecting to one of the three hosts. From the outside world, it’ll look like a valid cluster and we can test failover scenarios or other settings by simply bringing nodes down and seeing how the clients react. Final note On a final note, you might notice the KAFKA_offsets_topic_replication_factor: 3 setting. This setting defines the replication factor of the topic used to store the consumers offset. In the default case this is set to 1. So the consumer offsets for a particular topic will only be present on a single node. If that node goes down, consumers will lose track of where they are, since they can’t update the consumer offsets. This was the main problem in our case. By setting this to 3 we could safely cycle the nodes of the cluster, without it affecting the coonsumers.