Principles

The technical principles that guide this document are:

• Code maintenance and evolution - We should be able to maintain and evolve a code base over time. All developers should be able to follow sound and widespread standards. The system should be isolated between technical infrastructure decisions (database, external rest services, etc.) and the business logic domain.

• Cloud-native approach. The backend should be designed to operate on the cloud and take advantage of its surroundings. That topic means different things and keeps evolving. We will drill down what we mean by it in practical terms.

• Testability and quality. The backend must be testable and provide a good coverage of relevant test cases. The tests should be in line with modern spring boot test approaches.

• Spring Boot Native. The backend should follow and take advantage of spring boot auto configuration, containerization, tests, security, integrations, and extensions in a canonical way that is compatible with the framework's evolution without workarounds. Specific company extensions should consider how they play with the framework configuration model to extend it without compromising other features and integrations.

• I suggest following the principle of conventional overconfiguration and flexibility to override and adapt solutions. That means plugins developed by teams to use in Spring Boot can be opted in and out quickly and provide ways to be configured on specific behaviors and change (automatically or not) its configuration on different environments. That aligns with the Spring Boot Native principle and creates reusable components.

💡In the Spring Boot topic, I will explore how to combine AOP, Exception Handling, Meta Annotations, Conditional Beans, and Spring Boot Configuration Model to create powerful, reusable, clean, and sophisticated automation examples to plug-and-play in any Spring Boot application on the company, with minimal effort and flexibility in the hands of the consuming developers. An improvement on top of that is to distribute the configuration across multiple microservices; we will not cover that advanced scenario. However, we should be able to suggest ideas in the microservice architecture style that take advantage of all the code foundations mentioned in this document.

Code Maintenance and Evolution

To organize the code, we suggest the Clean Architecture structure in a single mono project by each microservice.

Clean Architecture is an architectural style created by Robert C. Martin that can be summarized in the following:

The clean architecture creates clear boundaries between business rules (stable and higher-level abstractions) and volatile technical details (lower-level abstractions). The main principle is the dependency rule: Source code dependencies must point only inward toward higher-level policies.

That means infrastructure details (databases, rest services, queues, etc..) are isolated from business logic and can be evolved, dismissed, or added in isolation. In fact, the clean code architecture should also have the following characteristics:

• Testable

• Independent UI Layer

• Independent of frameworks

• Independent of infrastructure like databases

On a high level, the structure of a spring boot project is as follows:

application
domain
infrastructure
presentation

Let's dive into each layer following a typical user request flow (from presentation to infrastructure)

The Presentation Layer

This layer will program the APIs and controllers. All external requests pass through this layer and are technology-agnostic. The most common way is through REST APIs. Still, it could be a console application, a GraphQL, GRPC, an async queue, or any other communication from the outside world to drive the application's behavior.

Talking about REST, there is an api/v0 folder inside the presentation. Each controller is versioned starting with version v0, which means it is under development and has no obligation to be stable or retrocompatible; each contract object is versioned inside the folder. When a stable version is published, it becomes v1, and another package is created, or the v0 is renamed. The v1 can evolve if it is retro-compatible with itself. If not, a v2 folder is created, and previous versions are still maintained in accordance with the team. There is no need to version at the class level in this approach.

These are the main things inside the presentation layer:

presentation
  api
    v0
      request
        CreatePersonRequest
      response
        CreatePersonResponse
      ExampleController.java
  config
    ObjectMapperConfig.java
    SwaggerConfig.java
  AppExceptionHandler.java

The clean architecture states: "Source code dependencies must point only inward toward higher-level policies." In practice, we can achieve this by isolating layers. The presentation layer depends only inward (depends on the application layer) and is isolated through the use of request and response objects. The convention is to use Js as the name prefix for requests and responses, which is optional and differentiates from domain classes when the names are very similar or equal (this is common while developing). What is not optional is the fact that these kinds of objects need to be created and transformed from/to the inwork layers.

The transformation technique is open to debate; some use mappers, some use static methods, and others use libraries to help, like the map struct library. Each team can use what it is most comfortable with. My only recommendation is to be consistent across the same microservice code base. The mapping is also used in the infrastructure layer; we will discuss it later.

The presentation layer will also have custom validation, pagination, and other API-related things.

Spring has a powerful programming model called Aspect Orienting Programming (AOP), and Spring Boot brings it to another level using the Conditional Beans feature. We could use these features to write clean and maintainable code, cleaning up many cross-cutting aspects of the application like logging, metrics, security, and auditing, and cleaning up significantly the controllers.

Exception Handling

Spring Boot has a canonical way of dealing with exceptions that is quite good.

The principle is to let the exceptions be thrown as specific runtime exceptions with the code and only catch each one in a single place. That is what the AppExceptionHandler does. Catching each one is not an accurate description, as we can catch groups and subclasses. We could also invert the logic of dependency to create a single catch and avoid having to treat each coupled with the AppExceptionHandler, making the exception treat itself (but that is just an idea open to be proved).

The UI will need specific codes to translate the messages and deal with errors. The Problems Details RFC standard could be interesting to study in other to standardize the responses from errors and validations.

The application layer

The presentation layer will get user inputs and delegate the execution to the application layer; the application layer can be considered as all application use cases. The key here is that the application layer will orchestrate the fulfillment of a flow or need of the user (be it a human or machine interacting with the application). The application layer can use many domain services and objects to fulfill its requests in a single transaction.

Speaking of transactions, this layer is a good candidate in most cases for controlling transactions, which means database transactions are initiated here.

When the controllers in the presentation layer access the application layer, they will convert the request to a domain object and pass it to it. Let's give an example:

application
   ExamplePersonUseCase.java
   ExamplePersonAdapterUseCase.java
presentation
  api
    v0
      ExamplePersonSetupController.java
      response
        PersonCreatedResponse.java
      request
        NewPersonRequest.java

@RestController
@RequiredArgsConstruct
@RequestMapping(path = "/api/v0/persons")
public class ExamplePersonSetupController {
    private final ExamplePersonUseCase examplePersonUseCase;

    @PostMapping()
    public PersonCreatedResponse createNewPerson(@Valid JsNewPersonRequest newPersonRequest) {
        return PersonCreatedResponse.of(examplePersonUseCase.createNewPerson(newPersonRequest.toDomain());
    }
}

The controller above is responsible for API-related tasks only (single responsibility principle).

The ExamplePersonUseCase is an interface with the contract between the layers:

public interface ExamplePersonUseCase {
    Person createNewPerson(Person person);
}

The ExamplePersonAdapterUseCase is an implementation. The adapter is a reference to ports and adapters. The interface is a port, and the adapter implements it. It generally has only one adapter, but it could have more, especially when dealing with different databases and persistence.

The pseudo implementation could be something like this:

@Service
@RequiredArgsConstructor
public class ExamplePersonAdaptUseCase implements ExamplePersonUseCase {
   private final ExamplePersonRepositoryService examplePersonRepositoryService;
   private final ExamplePersonRequisitsService examplePersonRequisitsService;


   @Override   
   @Transactional
   public Person createNewPerson(Person person) {
      examplePersonRequisitsService.checkIfCanBeAdded(person.getPassport());
      return examplePersonRepositoryService.createNewPerson(person);
   }
}

The Domain layer

The domain layer represents the business model. It overlaps with the data layer representation because it is common to have a representation as a persistence object (entity or table). But it is not the same. The business layer will represent the problem and can be modeled according to the system and problem. The only hard rule that Robert C. Martin talks about is:

The domain model should not directly depend on technology aspects.

That means if we need to persist or read data from a database, we should depend on an interface, not an implementation. Also we should minimize the use of external libraries in this layer.

Some tips:

• Avoid over-normalization of the domain layer. It will make it difficult and less performant to retrieve and persist information. It can also complicate data transformation.

• Create a rich model layer; do not use the domain layer only to carry data. You can reuse logic from the domain objects. For example, custom validation logic related to a single object can be added to the object itself.

• Prefer immutabibility.

• Remember: You still have the application layer to orchestrate multiple domain layer logic.

For example:

import lombok.Value;

import java.time.LocalDateTime;
import java.time.temporal.ChronoUnit;
import java.util.Locale;
import java.util.Map;
import java.util.Objects;

@Value
public class Person {

    private static final Map<String, Integer> LEGAL_AGE_IN_COUNTRIES = Map.of(
            Locale.US.getCountry(),
            21,
            Locale.UK.getCountry(),
            21,
            "BR",
            18
    );

    private String name;
    private LocalDateTime birthDay;

    private Address address;

    private Integer getLegalAgeInCountry(Locale.IsoCountryCode countryCode) {
        return LEGAL_AGE_IN_COUNTRIES.get(countryCode) == null ? 0 : LEGAL_AGE_IN_COUNTRIES.get(countryCode);
    }

    public int ageInYears() {
        return (int) ChronoUnit.YEARS.between(birthDay, LocalDateTime.now());
    }

    public boolean canSignIn(Locale.IsoCountryCode countryCode) {
        return ageInYears() >= getLegalAgeInCountry(countryCode) && Objects.equals(address.getCountryCode(), countryCode.name());
    }
}

This simple domain layer example shows that business logic related to the Person, like verifying whether it can sign in to the system, is performed in the domain object. The object is immutable (The @Value annotation).

If the domain layer exposes or consumes information, for example, loading the person from the database, it will depend on an interface. We call it a RepositoryService. The team can treat the repository service as a generic data access that does not rely on implementation; it could be a service, a relational database, or a nonrelational database. If you want, you could explicitly separate the concept of a data repository from an external REST service; just use a different name convention, likeExternalService.

For example:

import java.util.Optional;

public interface PersonRepositoryService {

    Optional<Person> findById(PersonId personId);
    Person save(Person person);
    List<Person> findAll();
}

Note: The service should consume domain objects and return domain objects. This means there will be a transformation in the infrastructure layer on the implementation side.

The Infrastructure Layer

The infrastructure layer performs every technology-related task, like database operations or external REST API

consumption. This layer typically implements the RepositoryService from the domain layer. We call this an Adapter. This concept is correlated with "Ports and Adapters" from the Alistar Cockburn Hexagonal Architecture reference article.

For example, this is the Spring Data JPA implementation of the PersonRespositoryService

@Repository
@RequiredArgsConstructor
public class PersonRepositoryAdapterImpl implements PersonRepositoryService {
    private final PersonRepository personRepository;

    @Override
    public Optional<Person> findById(PersonId personId) {
        return personRepository.findById(personId.id())
                .map(PersonEntity::toDomain);
    }

    @Override
    public Person save(Person person) {
        return personRepository.save(PersonEntity.fromDomain(person)).toDomain();
    }

    @Override
    public List<Person> findAll() {
        return personRepository.findAll()
                .stream()
                .map(PersonEntity::toDomain)
                .toList();
    }
}

The entity maps its contents from and to the domain layer in this example.

The infrastructure layer is divided into:

database
  entity
    AddressEntity.java
    PersonEntity.java
  repository
    PersonRepository.java
  PersonRespositoryAdapterImpl.java

This is technology-specific. You could have a rest folder in the infrastructure layer to communicate with REST API's

Check out this example repository for a complete code setup

Com isso podemos executar ferramentas exclusivas ou melhores no Linux enquanto mantemos nossa maquina principal com o windows.

Nesse primeiro video eu dou um overview de todas as ferramentas que vamos usar, e não há somente ferramentas de programação, mas também devops, segurança, documentação, gerenciamento de projetos e nerdices.

Vamos lá espero que gostem 👌

Take my wrong host file as an example:

# This file was automatically generated by WSL. To stop automatic generation of this file, add the following entry to /etc/wsl.conf:
# [network]
# generateHosts = false
127.0.0.1       localhost
127.0.1.1       Home-Z97.       Home-Z97

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

My machine is called Home-Z97, the entry "Home-Z97. Home-Z97" is wrong. You need to change it to: "Home-Z97"

Here is the correct full file:

# This file was automatically generated by WSL. To stop automatic generation of this file, add the following entry to /etc/wsl.conf:
# [network]
# generateHosts = false
127.0.0.1       localhost
127.0.1.1       Home-Z97

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

Save this and edit the file /etc/wsl.conf. Add the generatedHosts = false flag

[network]
enerateHosts = false

Restart WSL with the command in PowerShell: wsl --shutdown

This is a bizarre problem, and I came up with the solution by accident.

This solves the problem on two different machines. I hope that helps! 😊

Em uma das aulas nós refatoramos o código do controlador para separá-lo em camadas. O código está disponível em https://github.com/giovannicandido/aulas-fundatec-spring-rest-introducao/tree/aula-09

Nesse roteiro, vamos refatorar a camada de repository para persistir uma pessoa no banco de dados mysql. Vou assumir que os conceitos principais de bancos de dados já foram abordados (tabelas, selects, inserts, updates, filters, primary key, foreing key e joins)

Após ler esse roteiro você terá:

1. Mysql Instalado
2. Criado um banco de dados
3. Configurado o spring boot para ter suporte ao banco mysql e para conectar no banco criado
4. Refatorado o model e o repository de pessoa para ter os dados no banco
5. Testado a API

Vamos começar.

Instalando o mysql/mariadb

Há diversas formas de instalar o mysql, a melhor delas seria usando docker pois ele não somente permite usar o mysql mas milhares de outros serviços linux de maneira simples, além de ser a maneira mais profisional para rodar aplicações em ambientes de dev, homologação e produção.

Por isso vou abordar a instalação por ele, no entando vou deixar o link para instalação tradicional nas plataformas Windows e Linux, se a instalação por docker não der certo tente uma delas.

Nos laboratórios das aulas o mysql já vai estar instalado nas máquinas.

Também vou deixar links para se aprofundar no docker com mais tempo pois é um assunto mais extenso que vale apena o estudo, eu vou abordar apenas o necessário para ter o servidor mysql rodando que é o objetivo desse artigo.

Instalando o docker

Primeiramente precisamos do docker, mas o que é o docker?

O docker é um software que gerencia o que chamados de containers. Um conteiner é um processo de sistema isolado por meio de recursos do kernel do Linux. Esse processo possui sua rede virtual, sistema de arquivos isolado e uma imagem.

A imagem é um mecanismo que cria uma "foto" da instalação do software e de suas dependências, a imagem é imutável e formada por camadas que se assemelham a um commit do git.

Quando o docker roda um container, ele baixa a imagem de um registro e cria um processo através dela. A imagem serve então para distribuir o container.

O docker possui um registro central onde diversas imagens podem ser encontradas: o https://hub.docker.com

Um container se assemelha a uma máquina virtual (VM), porém é bem mais leve pois não possui a camada de sistema operacional nem toda emulação de hardware de uma VM.

Para uma definição mais detalhada de container acesse https://esr.rnp.br/administracao-de-sistemas/containers-docker-como-utilizar/

O docker é um dos runtimes para containers (temos também o containerd e o CRI-O) e foi o primeiro a popularizar essa tecnologia e ainda hoje é o mais popular.

É uma tecnologia que nasceu em cima de recursos do kernel do Linux, e um container compartilha o kernel do sistema host principal. No entando é possível instalar em Windows e MacOS através de VM's, mas não se preocupe você não precisa instalar uma máquina virtual com Linux temos uma alternativa muito mais simples.

Instalação do docker no Windows

Acesse: https://www.docker.com/products/docker-desktop/ Baixe o instalador e siga as instruções de instalação. É necessário estar em uma versão atual do Windows 10 ou 11 Deixe marcado a opção "use WSL2 instead of Hyper-V" Após a instalação ele vai iniciar com o sistema e vai estar no dock a direita com um icone que se assemelha a uma baleia. Para testar a instalação abra um terminal (de preferencia windows terminal com powershell) e digite:

docker version

Instalação no Linux

Vai depender da distribuição para o ubuntu siga as instruções em:

https://docs.docker.com/engine/install/ubuntu/

Nota: Use a docker engine e não o docker desktop para linux porque o docker desktop cria um VM, o docker engine roda nativamente e é mais performatico.

Criação do container com Mysql

Com o docker instalado é muito simples rodar um container mysql. Para isso basta um comando:

docker run -d --restart always --name mariadb -p 3306:3306 --env MARIADB_ROOT_PASSWORD=1234 mariadb:latest

Explicando o comando acima:

• docker run: Cria um container novo
• -d : Roda em background, ou seja libera a linha de comando
• --restart always : Diz ao docker que sempre deve iniciar o container com o systema, com isso nos não precisamos fazer um "docker start" para iniciar o container na mão
• --name mariadb : dá um nome para o container, se deixar sem nome depois só podemos referenciar o container por um id gerado automaticamente
• -p 3306:3306 : O container possui uma rede virtual e por padrão essa rede é inacessível para o host, somente entre os containers, então dizemos que a porta 3306 do host é mapeada para porta 3306 do container, assim acessando localhost 3306 acessamos o container
• --env : Configura variáveis de ambiente para o container. Nesse caso criamos uma variável que diz qual a senha de root do mysql
• mariadb:latest : mariadb é o nome da imagem que estamos rodando, latest é a versão. O docker interpreta latest como a ultima versão publicada. Na verdade o que está depois do : é uma tag, com isso podemos instalar versões específicas se quisermos.

Aqui vale uma distinção: O mariadb é o fork opensource do mysql. O mysql foi comprado pela oracle e de lá para cá ele existe em uma versão free e comercial, fizeram um fork dele para mantê-lo opensource.

Essas informações podem ser vistas no docker hub: https://hub.docker.com/_/mariadb

O que vai acontecer:

O docker vai baixar a imagem do mariadb e vai instanciar um container.

Poderemos acessar o mysql como de constume, por exemplo via linha de comando:

mysql -h localhost -u root -p

Uma coisa deve ser dita: O container mariadb armazena os seus dados em um sistema de arquivo interno e temporário. Caso o container seja removido os dados são perdidos. Um container é inicialmente projetado para ser efêmoro, isso é, morrer e não carregar seus dados consigo. A idéia é que ele possa ser instanciado instantaneamente em outra máquina. Abordagens são necessárias para manter o estado (dados) de um container. Uma delas é o mapeamente de volume

Se você fizer um

docker stop mariadb
docker rm mariadb

Todos os dados serão perdidos.

Acessando o mysql

Há diversas formas. Eu uso a nativa do Intellij mas ela é paga. Recomendo o dbeaver pois nele pode-se conectar a diversos tipos de bancos (como o postgresql) e está disponível para todas as plataformas (Linux, Mac e Windows)

Baixe e instale-o em: https://dbeaver.io/download/

Escolha o mariadb como base de dados:

Preencha os dados de host (localhost) porta (3306) usuário (root) e senha (1234)

Expanda o servidor, clique com o botão direito em cima de Databases e crie uma nova base de dados com nome de lpII

A partir daqui você pode criar tabelas, fazer inserts entre outros mas vamos configurar o Spring para criar as tabelas para agente.

Refatorando a aplicação para usar banco de dados

Configurando a Aplicação para Conectar ao Banco

Vamos configurar a aplicação para usar o spring data jpa como framework de acesso a dados.

Edite o arquivo pom.xml na raiz do projeto e adicione as seguintes dependencias:

 <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-data-jpa</artifactId>
  </dependency>
  <dependency>
      <groupId>org.mariadb.jdbc</groupId>
      <artifactId>mariadb-java-client</artifactId>
      <version>3.1.0</version>
  </dependency>

O starter JPA é a depedencia para trabalhar com dados. O mariadb-java-client é o driver do banco que vamos conectar.

Dê um refresh no maven do projeto

Agora edite o arquivo src/main/resources/application.properties

Configure o spring para conectar no banco com as propriedades:

spring.datasource.url=jdbc:mariadb://localhost/lpII
spring.datasource.username=root
spring.datasource.password=1234

Rode o projeto. Se o spring não apresentar erros a conexão com o banco foi estabelecida. Exemplo:

Vamos configurar o hibernate JPA para gerar as tabelas com base nos modelos para agente: adicione a linha abaixo no application.properties

spring.jpa.hibernate.ddl-auto=update

Atualizando o modelo de dados

Agora precisamos de uma tabela. Edite o arquivo model/Pessoa e adicione as minimas anotações:

O restante da classe deixe como está. Rode o projeto novamente e veja se uma tabela pessoa foi criada

Explicando as anotações:

• @ Entity - Colocada na classe diz que ela é uma entidade, ou seja, uma tabela no banco de dados
• @ Id - Toda entidade deve possui um ID que a identifica unicamente na tabela (primary key), O id então será a propriedade id do tipo Long @ GeneratedValue - Instrui o hibernate a gerar essa propriedade usando alguma estratégia (auto increment, sequence, etc). No caso o strategy é IDENTITY que significa que o banco vai gerar para agente um numero incremental.

Para o restante das propriedades o hibernate vai assumir o padrão.

• Para cada propriedade ele vai criar uma coluna correspondente, com o tipo correspondente. Exemplo: String é mapeado para varchar(255) e LocalDate é mapeado para date
• O nome da coluna é o nome da propriedade

Podemos sobrescrever os default com outras anotações. Por exemplo para alterar o tamanho de uma coluna ou o seu nome:

A propriedade spring.data.hibernate.ddl-auto=update tentará atualizar o banco, no entando há casos em que ela não consegue. Altere temporariamente o valor para create-drop reinicie o spring e a tabela será criada novamente com a coluna nome em varchar(60)

Modificando o repository

Vamos alterar o repository de pessoa para que as operações sejam feitas por ele. Temos duas opções:

1. Criar um repository com a API do spring data e quebrar o contrato com o service, fazendo com que tenhamos que refatorar todos os metodos do service
2. Implementar uma interface com o contrato entre service e repository

Vamos fazer o segundo por ser mais elaborado. Mas note que em um projeto que começe com spring data isso não seria necessário pois já se utilizaria as API dos mesmo

Renomeie a classe PessoaRepository para PessoaRepositoryImpl

No intellij clique com o botão direito em qualquer metodo da classe e vá em refactor > extract interface.

De o nome da interface de PessoaRepository

Selecione todos os metodos publicos

Crie e quando for perguntado se deseja analisar e fazer o replace das partes que usam PessoaRepositoryImpl para PessoaRepository diga sim.

Verifique se a classe PessoaService agora depende de PessoaRepository (interface) Verifique se a classe teste está ok. Atualmente ela não faz nada então os testes devem compilar e passar (vamos abordar testes de spring em outro momento)

Crie uma nova interface PessoaRepositorySpring

Extenda de JpaRepository e passe os generic Pessoa e Long:

Esse seria um repository padrão do spring. Explicando:

Ao estender de JpaRepository herdamos todos os metodos necessários para se trabalhar com o banco. O spring irá injetar essa classe e criar a implementação para agente. O primeiro generic, é o tipo de Entidade que o repository trabalha, no caso Pessoa, o segundo generic é o tipo do ID que pessoa tem, no caso Long

Temos duas opções para não quebrar a API.

A primeira opção seria criar uma nova classe de implementação de PessoaRepository e usar o PessoaRepositorySpring dentro dela:

Mas isso gera uma classe a mais sem necessidade. A segunda opção é que podemos fazer com que a propria interface PessoaRepositorySpring implemente os metodos usando o recurso de default implementation das interfaces java:

Porém há um problema. O método findById da interface PessoaRepository não é compativel e não pode ser sobrescrito pelo metodo findById de JpaRepository fazendo com que haja um problema de compilação. Para isso a melhor forma é fazer com que o retorno de PessoaRepository seja compatível com ele, ou renomear o método.

Vamos renomear o metodo para que não quebre o retorno esperado pelo Service (lembre-se estamos fazendo isso tudo para não refatorar o service ou seja não alterar a camada superior com uma implementação nova da camada repository)

Faça um refactoring renomentado PessoaRepository findById para findWithId e adicione a implementação em PessoaRepositorySpring:

Tente rodar a aplicação.

Temos um erro de compilação que diz: "reference to deleteById is ambiguous" Isso acontece porque tanto o PessoaRepositorySpring, quanto PessoaRepository declaram deleteById e PessoaRepositorySpring extende PessoaRepository.

Para resolver renomeie o metodo em PessoaRepository e implemente em PessoaRepositorySpring:

Tente rodar a aplicação:

Spring vai falhar com uma mensagem:

Parameter 0 of constructor in br.org.fundatec.lp2.demorest.service.PessoaService required a single bean, but 3 were found:

• pessoaRepositoryImpl: defined in file [C:\Users\giova\Projects\personal\aulas\fundatec\aula09\demo-rest\target\classes\br\org\fundatec\lp2\demorest\repository\PessoaRepositoryImpl.class]
• pessoaRepositoryImplSring: defined in file [C:\Users\giova\Projects\personal\aulas\fundatec\aula09\demo-rest\target\classes\br\org\fundatec\lp2\demorest\repository\PessoaRepositoryImplSring.class]
• pessoaRepositorySpring: defined in br.org.fundatec.lp2.demorest.repository.PessoaRepositorySpring defined in @EnableJpaRepositories declared on JpaRepositoriesRegistrar.EnableJpaRepositoriesConfiguration

Isso acontece porque agora temos 3 implementação possíveis para PessoaRepository e o spring não sabe qual injetar.

A forma mais simples de resolver isso é setar uma das implementações como @Primary isso vai fazer com que o spring use-a.

Primeiramente vamos setar como @Primary nossa antiga implementação de repository em memória com listas:

Reinicie e teste inserir alguma coisa usando o Insomnia REST

Verifique que a API retorna itens usando o GET pessoas, mas no dbeaver não vai haver registros

No dbeaver vá em Sql Editor > Open SQL Console

Digite a query:

select * from pessoa;

Agora troque a implementação para uma das classes de repository que usam o Spring JPA (PessoaRepositoryImplSpring ou PessoaRepositorySpring), adicionando o @Primary em outra classe (não se esqueça de removê-lo de PessoaRepositoryImpl)

Repita o teste e rode a query novamente. Deve encontrar os registros:

Conclusão

Nos refatoramos o repository para salvar no banco de dados os dados de pessoa. No final acabamos com 5 classes no repository:

• PessoaIndex - Auxiliar para trabalhar com a lista em memória
• PessoaRepository - Nossa interface principal da qual PessoaService depende
• PessoaRepositorySpring - Interface que usa o spring para ir no banco de dados. Essa é a classe principal para conexão com o banco
• PessoaRepositoryImpl - Nossa implementação de lista em memória
• PessoaRepositoryImplSpring - Uma implementação intermediaria como alternativa para manter a compatibilidade com a interface do service.

Na próxima aula vamos criar um novo endpoint, ele nele vamos ter apenas uma interface para repository, a interface principal do spring.

O código para essa aula está disponível em: https://github.com/giovannicandido/aulas-fundatec-spring-rest-introducao/tree/aula-11

Referências

Seguem alguns links para estudarem os assuntos:

Instalação do mariadb:

https://mariadb.org/download

Documentação sobre Docker:

https://docs.docker.com/get-started/ https://blog.geekhunter.com.br/docker-na-pratica-como-construir-uma-aplicacao/

Documentação Spring Data e JPA

https://www.baeldung.com/the-persistence-layer-with-spring-data-jpa https://www.treinaweb.com.br/blog/iniciando-com-spring-data-jpa

https://www.alura.com.br/apostila-java-web/uma-introducao-pratica-ao-jpa-com-hibernate https://www.devmedia.com.br/jpa-e-hibernate-acessando-dados-em-aplicacoes-java/32711 Vários Tutoriais: https://www.javaguides.net/p/jpa-tutorial-java-persistence-api.html

Apresentação sobre JPA https://1drv.ms/p/s!AoHvV-Rb6N9Qg9YV8286A29SOLJnbw?e=51hdJh

We will not cover ansible automation platform, we will talk about ansible core and modules, because the automation platform is a enterprise tool that offer many features to control and execute ansible scripts in any environment you have. It's good for big teams when you need more control and contribution over the playbooks you run. Also is a way more complex software.

The official ansible site is a little confuse to starters, because it show documentation primary focus on the Red Hat Ansible Automation Platform, and as I mention this software is focused on enterprise customers. The ansible core is opensource and free to use and is much more simpler to start with.

Core Concepts

How ansible automate the tasks necessary for Linux provision? To answer that question lets talk about ansible core

The ansible core is collection of CLI tools, the Ansible language and a architectural framework that allows extensions though ansible collections.

The ansible language is based on YAML to express the desired state of the machines you are controlling. This YAML is interpreted by python under the covers that implements the functionality with build in modules or extended modules installed with ansible-galaxy.

Ansible connects to targets through SSH and use an agentless approach to function. This has advantages over agent systems because you don't have to deal with daemons installed in the machines, this leads to a more straightforward approach, ansible only requires SSH and python installed in the target machine, which most Linux distros bring by default. The disadvantages of this approach that I can bring to table are a less performant operation, because ansible has to connect to each host, the parallelization is somewhat limited but possible (check ansible serial command and this post for parallel tasks.

To understand ansible as a minimum we need to cover this topics:

• The ansible inventory
• Ad Hoc Commands
• Ansible Playbooks and Tasks
• Modules and Collections
• Handlers
• Roles and the recommend layout for playbooks

We will talk about then later.

Installing Ansible

Ansible works best on Linux host, Windows is not supported. If you are in Windows I recommend installing in you WSL2 distro.

You need an minimal python 3.8 installed. In ubuntu 20.04.4 is already installed

I use arch Linux so I will cover how to install in arch as well.

In ubuntu do not install the package ansible, because it's old.

Run the follow commands:

python3 -m pip -V
# If pip is not installed
sudo apt install python3-pip
python3 -m pip install --user ansible # install ansible for your current user

Ansible will be installed for your current user. You need ~/.local/bin on your PATH. To do this edit ~/.bashrc (or ~/.zshrc in zshell) and add the line:

export PATH=~/.local/bin:$PATH

Reopen your terminal to take effect and test the installation with:

ansible --version

On Arch Linux the community package is up to date so you can just:

sudo pacman -Sy ansible

Upgrading ansible

To upgrade in ubuntu run the command:

python3 -m pip install --upgrade --user ansible

In Arch Linux:

sudo pacman -Syu ansible

Getting Started

Building an Inventory

An inventory is where you target your Linux machines and groups then so you can run commands and playbooks on it. It provides system information like username and network ip address reducing the number of command line options you need to specify.

The managed node (a remote system or host, that ansible controls) can be specified in inventories or in the file /etc/ansible/hosts. For a example of the hosts file see the file below:

[myvirtualmachines]
192.0.2.50
192.0.2.51
192.0.2.52

It creates a group of machines called myvirtualmachines and adds 3 ip address to it.

When connecting to the host you need to specify the -u option for the username (if is different than current logged user). To check the host run the command:

ansible all --list-hosts

You should see the output:

hosts (1):
192.0.2.50
192.0.2.51
192.0.2.52

To ping in the host and check connection run the command:

ansible all -m ping

Lets see how to use inventories, which are a more powerful way to catalog your managed nodes.

Create a folder ansible-test to sum up the files for this article, all files will be created inside it.

Create a file named inventory.yaml (you can use ini files as well)

Add the following example

virtualmachines:
  hosts:
    vm01:
      ansible_user: vagrant
      ansible_host: 192.168.10.4
    vm02:
      ansible_user: vagrant
      ansible_host: 192.168.10.5
    vm03:
      ansible_user: vagrant
      ansible_host: 192.168.10.6

Verify your inventory:

ansible-inventory -i inventory.yaml --list

I'm using a few options here because I'm using hyper-v and vagrant to spin up the machines. Lets explain them:

• ansible_ssh_private_key_file: Use a private key file that has generated by vagrant in machine creation. This is optional
• ansible_user: the user that ansible will connect to
• ansible_host: the ip address of the VM

Lets ping your inventory but first you need some Ubuntu server Linux machines, spin around some in a virtualized environment and make sure you can connect with SSH. If you want use Hyper-v and want overcome some problems with it, check my blog post about Hyper-V, Vagrant and Ansible

SSH will check host key on first connection effectively stopping the connection on first because we need to say 'yes' to each host. To avoid that. Add the following to your /etc/ansible/ansible.cfg file:

[ssh_connection]
ssh_args = -o StrictHostKeyChecking=accept-new

Now we can:

ansible virtualmachines -m ping -i inventory.yaml

We receive something like this as response:

vm03 | SUCCESS => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/bin/python3"
},
"changed": false,
"ping": "pong"
}
vm02 | SUCCESS => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/bin/python3"
},
"changed": false,
"ping": "pong"
}
vm01 | SUCCESS => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/bin/python3"
},
"changed": false,
"ping": "pong"
}

Your communication is set up. Lets see how we can run ad-hoc commands:

Running Ad-Hoc Commands

Ad hoc commands is a way to run just a few commands in the target hosts. We can use that to for example, update all host packages or to check the contents of a file in all hosts. This are not as powerfull as playbooks but are usefull.

Lets get started by checking the contents of a file in all machines:

Run the command;

ansible virtualmachines -m shell -a "cat /etc/resolv.conf" -i inventory.yaml

You should see the output:

vm02 | CHANGED | rc=0 >>
# This is /run/systemd/resolve/stub-resolv.conf managed by man:systemd-resolved(8).
# Do not edit.

# This file might be symlinked as /etc/resolv.conf. If you're looking at
# /etc/resolv.conf and seeing this text, you have followed the symlink.

# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.

# Run "resolvectl status" to see details about the uplink DNS servers
# currently in use.

# Third party programs should typically not access this file directly, but only
# through the symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a
# different way, replace this symlink by a static file or a different symlink.

# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 127.0.0.53
options edns0 trust-ad
search .
vm01 | CHANGED | rc=0 >>
# This is /run/systemd/resolve/stub-resolv.conf managed by man:systemd-resolved(8).
# Do not edit.

# This file might be symlinked as /etc/resolv.conf. If you're looking at
# /etc/resolv.conf and seeing this text, you have followed the symlink.

# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.

# Run "resolvectl status" to see details about the uplink DNS servers
# currently in use.

# Third party programs should typically not access this file directly, but only
# through the symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a
# different way, replace this symlink by a static file or a different symlink.

# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 127.0.0.53
options edns0 trust-ad
search .
vm03 | CHANGED | rc=0 >>
# This is /run/systemd/resolve/stub-resolv.conf managed by man:systemd-resolved(8).
# Do not edit.

# This file might be symlinked as /etc/resolv.conf. If you're looking at
# /etc/resolv.conf and seeing this text, you have followed the symlink.

# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.

# Run "resolvectl status" to see details about the uplink DNS servers
# currently in use.

# Third party programs should typically not access this file directly, but only
# through the symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a
# different way, replace this symlink by a static file or a different symlink.

# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 127.0.0.53
options edns0 trust-ad
search .

Which shows the output of the command for each machine.

Now lets update all packages in the hosts:

ansible virtualmachines -m shell -a "apt update && apt upgrade -y" --become -i inventory.yaml

We use the --become option to run the command as root. We also need to add -y to the apt upgrade because we do not has access to a interactive shell so we say yes to everything.

This command takes a few time to execute, and the result is the output of the apt command. If you add --ask-become-pass or -K, Ansible will prompt your for the password to use for privilege escalation.

Let me explain the options of this command:

• virtualmachines: The first parameter is a selector for the machines in inventory. In this example we are selectiong all machine in group "virtualmachines".
• -m: selects the module that ansible will execute. Ansible is a righly modular tools and even the core is modular. In this case we are selecting the "shell" module that executes commands in the target machine.
• -a: Module parameters. We provide de command for the module shell
• --become: Run the module as root using privilege escalation (sudo)
• -i: Select the inventory

We use the generic shell module to run a command. But we can use any module, in this case the use of a package module like yum or apt is more powerfull to install a specific package because is indempontent (it will not run if not needed by checking the package state.

To install a package use the command:

ansible virtualmachines -m ansible.builtin.apt -a "name=httpd state=present"

This command will ensure the package name httpd is installed. If you want to make sure is updated use:

ansible virtualmachines -m ansible.builtin.apt -a "name=httpd state=latest"

Another good use case for ad hoc command is to manage files using the built in file and copy module. File module allows changing owernership and permissions on files, or create directories:

# Copy a file from host to target
ansible virtualmachines  -m ansible.builtin.copy -a "src=/etc/hosts dest=/tmp/hosts"
# change permission of file
ansible virtualmachines -m ansible.builtin.file -a "dest=/srv/foo/a.txt mode=600"
ansible virtualmachines -m ansible.builtin.file -a "dest=/srv/foo/b.txt mode=600 owner=mdehaan group=mdehaan"
# creates a directory
ansible virtualmachines -m ansible.builtin.file -a "dest=/path/to/c mode=755 owner=mdehaan group=mdehaan state=directory"
# deletes a file or directory
ansible virtualmachines -m ansible.builtin.file -a "dest=/path/to/c state=absent"

Ansible also has the built in module user to manage users and groups. Check it's documentation

Another useful thing to do in ad hoc commands is to manage services using the service module. Examples:

Ensure a service is started on all webservers:

$ ansible webservers -m ansible.builtin.service -a "name=httpd state=started"

Alternatively, restart a service on all webservers:

$ ansible webservers -m ansible.builtin.service -a "name=httpd state=restarted"

Ensure a service is stopped:

$ ansible webservers -m ansible.builtin.service -a "name=httpd state=stopped"

Selecting Targets

There is a few ways you could select targets in the inventory. You can use a pattern to do that:

You can also combine this expressions. For example webservers:dbservers:&staging:!phoenix

There is a wildcard option for FQDNS or IP address, as long as the hosts are named in your inventory by FQDN or IP address:

192.0.*
*.example.com
*.com

You can mix wildcard patterns and groups at the same time:

one*.com:dbservers

Limitations of patterns

Patterns depend on inventory. If a host or group is not listed in your inventory, you cannot use a pattern to target it. If your pattern includes an IP address or hostname that does not appear in your inventory, you will see an error like this:

[WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: Could not match supplied host pattern, ignoring: *.not_in_inventory.com

Your pattern must match your inventory syntax. If you define a host as an alias:

atlanta:
  host1:
    http_port: 80
    maxRequestsPerChild: 808
    host: 127.0.0.2

you must use the alias in your pattern. In the example above, you must use host1 in your pattern. If you use the IP address, you will once again get the error:

[WARNING]: Could not match supplied host pattern, ignoring: 127.0.0.2

Using regexes in patterns

You can specify a pattern as a regular expression by starting the pattern with ~:

~(web|db).*\.example\.com

Patterns and ad-hoc commands

You can change the behavior of the patterns defined in ad-hoc commands using command-line options. You can also limit the hosts you target on a particular run with the --limit flag.

• Limit to one host

$ ansible -m [module] -a "[module options]" --limit "host1"

• Limit to multiple hosts

$ ansible -m [module] -a "[module options]" --limit "host1,host2"

• Negated limit. Note that single quotes MUST be used to prevent bash interpolation.

$ ansible -m [module] -a "[module options]" --limit 'all:!host1'

• Limit to host group

$ ansible -m [module] -a "[module options]" --limit 'group1'

For more information about selection pattern check the docs: https://docs.ansible.com/ansible/latest/user_guide/intro_patterns.html

Ansible Playbooks and Tasks

This is where the fun starts, a playbook is a repeatable, re-usable configuration management and multi-machine deployment system. Is a good idea to version control your playbooks so you can backup and share with others.

They are expressed as YAML format with a easy syntax. The playbook runs from top to bottom. Within each play, tasks also run in order from top to bottom. Playbooks can also include each other and orchestrate multiple machine deployments, one for webservers another for databases and other for network infrastructure, and so on. As a minimum each play defines two things:

• the managed nodes to target, using the pattern we saw early
• at least one task to execute

Many Ansible modules have idempotency built in (but not all of them) that means ansible check the desired state has already been achieved and exit without performing any actions if that state has been achieved. I you repeat the playbook execution or the tasks that use idempotent modules the final state is not changed.

Lets see an example of playbook:

---
- name: Update web servers
  hosts: webservers
  become: true
  vars:
    httpd_config_folder: /etc/apache2
    vhost_domain: example.local
    proxy_timeout: 60
  tasks:
  - name: Ensure apache is at the latest version
    ansible.builtin.apt:
      name: apache2
      state: latest
  - name: Creates the html directory for site
    ansible.builtin.file:
      dest: '/var/www/vhosts/{{ vhost_domain }}/public_html'
      state: directory
  - name: Create the site index.html
    ansible.builtin.template:
      src: ./templates/index.html.j2
      dest: '/var/www/vhosts/{{ vhost_domain }}/public_html/index.html'
  - name: Config virtual host for site
    ansible.builtin.template:
      src: ./templates/site-vhost.conf.j2
      dest: '{{ httpd_config_folder }}/sites-available/{{ vhost_domain }}.conf'

    notify:
      - Restart apache
  - name: Disables apache default vhost which conflicts with others
    ansible.builtin.command: "a2dissite 000-default.conf"
    args:
      removes: '{{ httpd_config_folder }}/sites-enabled/000-default.conf'
    notify:
      - Restart apache
  - name: Enable VHost for Domain {{ vhost_domain }}  in apache
    ansible.builtin.command: "a2ensite {{ vhost_domain }}.conf"
    args:
      creates: '{{ httpd_config_folder }}/sites-enabled/{{ vhost_domain }}.conf'
    notify:
      - Restart apache
  - name: Ensure Apache2 is Started
    ansible.builtin.service:
      name: apache2
      state: started
  handlers:
    - name: Restart apache
      ansible.builtin.service:
        name: apache2
        state: restarted

- name: Update db servers
  hosts: databases
  become: true

  tasks:
  - name: Ensure postgresql is at the latest version
    ansible.builtin.apt:
      name: postgresql
      state: latest
  - name: Ensure that postgresql is started
    ansible.builtin.service:
      name: postgresql
      state: started

This playbook do a lot of things:

1. On webservers group install apache2 and make sure is updated
2. Creates folders for a vhost site public html
3. Creates a index.html for the vhost
4. Configures the vhost using a template that uses some variables like proxy_timeout and the vhost_domain
5. Disables the apache default vhost which conflicts with all others, if the files exists. When changed notify a handler (trigger), that will restart apache
6. Enables the vhost site. When the symlink file is created in the enabled sites it will not run again. When changed notify a handler that will restart apache
7. Ensure apache is started
8. Declares a handler for apache restarts. Handlers are executed at the final of playbooks, only if the notification of some task is trigged
9. Install postgresl on databases group and ensure it's updated
10. Ensure postgresql server is started

The inventory for this groups of machines could be something like:

virtualmachines:
  hosts:
    vm01:
      ansible_user: vagrant
      ansible_host: 192.168.10.4
    vm02:
      ansible_user: vagrant
      ansible_host: 192.168.10.5
    vm03:
      ansible_user: vagrant
      ansible_host: 192.168.10.6
webservers:
  hosts:
    vm01:
    vm02:
databases:
  hosts:
    vm03:

And the templates are as follow:

index.html.j2 template

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <meta http-equiv="X-UA-Compatible" content="ie=edge">
    <title>Ansible Getting Started</title>
  </head>
  <body>
    <main>
        <h1>Welcome to {{ vhost_domain }}</h1>  
    </main>
  </body>
</html>

site-vhost.conf.j2 template:

<VirtualHost *:80>
    ServerName {{ vhost_domain }}
    Timeout {{ proxy_timeout }}
    KeepAliveTimeout {{ proxy_timeout }}
    MaxKeepAliveRequests 0

    ServerAlias www.{{ vhost_domain }}

    DocumentRoot /var/www/vhosts/{{ vhost_domain }}/public_html

    <Directory /var/www/vhosts/{{ vhost_domain }}/public_html>
        Options Indexes FollowSymLinks MultiViews
        AllowOverride All
    </Directory>



    CustomLog /var/log/apache2/{{ vhost_domain }}-access.log combined

    ErrorLog /var/log/apache2/{{ vhost_domain }}-error.log
    # Possible values include: debug, info, notice, warn, error, crit,

    # alert, emerg.

    LogLevel warn

</VirtualHost>

The .j2 extension are for Jinja templates which are the sintax for them.

If you run this playbook on the machines, you can check the html by adding a DNS local hostname line in your hosts file (/etc/hosts) like the one below

192.168.10.5 example.local

Any IP of webservers group should work.

You could check the ip with curl:

curl example.local

Ansible Modules and Collections

The module is a reusable binary that ansible copies and executes on each node (when needed) to accomplish the action defined in each task. The modules are specific, for example: administering users on specific database, managing VLAN interfaces on a network device. Each task invokes a single module, and with playbooks you invoke several different modules..

Ansible >= 2.0 recommends that you use the full qualified name of the module (for example ansible.builtin.copy), because of conflicts with single names.

The latest version of Ansible documentation talks about collections, which are a standard way to distribute playbooks, plugins, roles, and modules.

You can install collections with ansible-galaxy command or with a requirements file which you can distribute with your playbook codes.

The default installation of ansible comes with some collections pre-installed. You can check then with the command ansible-galaxy collection list

There is a lot you can do with the ansible collections, like referencing then in playbooks, importing playbooks distributed in collections and many more. Check the documentation: https://docs.ansible.com/ansible/latest/user_guide/collections_using.html#using-collections-in-a-playbook

We already saw a basic usage of the builtin collection in your previous playbook example.

To a list of all modules check this page: https://docs.ansible.com/ansible/latest/collections/index_module.html

Some interesting collections are:

• amazon.aws - Manages AWS resources
• community.aws - Manages AWS resources
• ansible.builtin - The builtin collection with many usefull tools
• ansible.posix - Tool to manage ACL, add SSH authorized keys, synchronize file with rsync, and more
• azure.azcollection - Manages Azure resources
• community.azure - Manages Azure resources
• community.digitalocean - Manages digital ocean resources
• community.dns - Manages DNS zones on some providers.
• community.docker - Manages docker containers
• community.general - Many community modules for tools like keycloak, influxdb, FreeIPA and others, Java
• community.google - Manages Google cloud resources
• google.cloud - Manages Google cloud resources (more complete)
• community.mysql - Manages Mysql databases
• community.postgresql - Manages Postgresql databases
• community.rabbitmq - Manages RabbitMQ exchanges, bindings, queues and other parameters.
• community.vmware - Manages VMware virtualized infrastructure.
• kubernetes.core - Core functionality for Kubernetes clusters

And many more collections. You can also search in the galaxy repository: https://galaxy.ansible.com/

Handlers

In the previous playbook we saw a handler in action. It execute a restart in apache only when need, or in other words, only when one of the tasks changes and trigger the actions. We need to declare the handler and to notify then. Here is another example:

tasks:
- name: Template configuration file
  ansible.builtin.template:
    src: template.j2
    dest: /etc/foo.conf
  notify:
    - Restart apache
    - Restart memcached

handlers:
  - name: Restart memcached
    ansible.builtin.service:
      name: memcached
      state: restarted

  - name: Restart apache
    ansible.builtin.service:
      name: apache
      state: restarted

This example notify two handlers: Restart memcached and Restart apache

You can also name the handler different and listen to the same event (notify). In the follow example all two handlers will be executed:

tasks:
  - name: Restart everything
    command: echo "this task will restart the web services"
    notify: "restart web services"

handlers:
  - name: Restart memcached
    service:
      name: memcached
      state: restarted
    listen: "restart web services"

  - name: Restart apache
    service:
      name: apache
      state: restarted
    listen: "restart web services"

Controlling when handlers run

By default handlers run after all tasks in a particular play have been completed. With this approach handlers runs only once, regardless of how many tasks notify it.

If you need to run handlers before the end of the play, add a task to flush the handlers after the one you want using the meta module. Example:

tasks:
  - name: Some tasks go here
    ansible.builtin.shell: ...

  - name: Flush handlers
    meta: flush_handlers

  - name: Some other tasks
    ansible.builtin.shell: ...

For more information about handlers check the documentation: https://docs.ansible.com/ansible/latest/user_guide/playbooks_handlers.html

Roles and the recommend layout for playbooks

You can separate your playbooks into roles, and then apply roles to target nodes. This way you separate in another file the roles and can reuse them. Roles also store defaults, handlers, variables and tasks in separate directories, instead of a single long document. But first lest talk about the recommended layout to organize ansible playbooks and it will become more clear how roles works.

The directory layout recommended in the Ansible documentation is:

production                # inventory file for production servers
staging                   # inventory file for staging environment

group_vars/
   group1.yml             # here we assign variables to particular groups
   group2.yml
host_vars/
   hostname1.yml          # here we assign variables to particular systems
   hostname2.yml

library/                  # if any custom modules, put them here (optional)
module_utils/             # if any custom module_utils to support modules, put them here (optional)
filter_plugins/           # if any custom filter plugins, put them here (optional)

site.yml                  # master playbook
webservers.yml            # playbook for webserver tier
dbservers.yml             # playbook for dbserver tier

roles/
    common/               # this hierarchy represents a "role"
        tasks/            #
            main.yml      #  <-- tasks file can include smaller files if warranted
        handlers/         #
            main.yml      #  <-- handlers file
        templates/        #  <-- files for use with the template resource
            ntp.conf.j2   #  <------- templates end in .j2
        files/            #
            bar.txt       #  <-- files for use with the copy resource
            foo.sh        #  <-- script files for use with the script resource
        vars/             #
            main.yml      #  <-- variables associated with this role
        defaults/         #
            main.yml      #  <-- default lower priority variables for this role
        meta/             #
            main.yml      #  <-- role dependencies
        library/          # roles can also include custom modules
        module_utils/     # roles can also include custom module_utils
        lookup_plugins/   # or other types of plugins, like lookup in this case

    webtier/              # same kind of structure as "common" was above, done for the webtier role
    monitoring/           # ""
    fooapp/               # ""

Alternatively you can put each inventory file with group_vars and host_vars in a separated directory. This is useful if your group_vars and host_vars don't have hat much in common in different environments. The layout could look like this:

inventories/
   production/
      hosts               # inventory file for production servers
      group_vars/
         group1.yml       # here we assign variables to particular groups
         group2.yml
      host_vars/
         hostname1.yml    # here we assign variables to particular systems
         hostname2.yml

   staging/
      hosts               # inventory file for staging environment
      group_vars/
         group1.yml       # here we assign variables to particular groups
         group2.yml
      host_vars/
         stagehost1.yml   # here we assign variables to particular systems
         stagehost2.yml

library/
module_utils/
filter_plugins/

site.yml
webservers.yml
dbservers.yml

roles/
    common/
    webtier/
    monitoring/
    fooapp/

In the roles directory you separate tasks, default variables, templates, and handlers and them apply this using targets in your playbook. Lets change your example of installing apache to follow this best practice.

The folder structure will be:

inventories
  local-hyperv
    group_vars
      webservers.yaml  
    hosts.yaml
  production
    group_vars
      webservers.yaml
    hosts.yaml
roles
  postgresl
    tasks
      main.yaml
  webserver
    handlers
      main.yaml
    tasks
      main.yaml
    templates
      index.html.j2
      site-vhost.conf.j2
    vars
      main.yaml
databases.yaml
site.yaml
webservers.yaml

Inside your inventory folder we create two environments: local-hyperv and production. We will override some variables to customize the playbooks in the group_vars folder, that overrides variables for groups of servers (there is also a host_vars folder that could be used to customize one host). In this example we create a webservers.yaml which overrides the variables for the webservers group.

local-hyperv webservers.yaml:

vhost_domain: example.local
proxy_timeout: 60

production webservers.yaml:

vhost_domain: example-prod.local
proxy_timeout: 80

Your hosts.yaml configuration differs only in the ip address of the servers.

local-hyperv hosts.yaml:

virtualmachines:
  hosts:
    vm01:
      ansible_user: vagrant
      ansible_host: 192.168.10.4
    vm02:
      ansible_user: vagrant
      ansible_host: 192.168.10.5
    vm03:
      ansible_user: vagrant
      ansible_host: 192.168.10.6
webservers:
  hosts:
    vm01:
    vm02:
databases:
  hosts:
    vm03:

production hosts.yaml:

virtualmachines:
  hosts:
    vm01:
      ansible_user: vagrant
      ansible_host: 192.168.10.7
    vm02:
      ansible_user: vagrant
      ansible_host: 192.168.10.8
    vm03:
      ansible_user: vagrant
      ansible_host: 192.168.10.9
webservers:
  hosts:
    vm01:
    vm02:
databases:
  hosts:
    vm03:

We then create your roles folder, the postgresql roles just setup some tasks:

roles/postgresql/tasks/main.yaml

- name: Ensure postgresql is at the latest version
  ansible.builtin.apt:
    name: postgresql
    state: latest
- name: Ensure that postgresql is started
  ansible.builtin.service:
    name: postgresql
    state: started

The webserver role is a little more complete it separated in handlers, tasks, templates and vars:

roles/webserver/handlers/main.yaml

- name: Restart apache
  ansible.builtin.service:
    name: apache2
    state: restarted

roles/webserver/templates/index.html.j2

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <meta http-equiv="X-UA-Compatible" content="ie=edge">
    <title>Ansible Getting Started</title>
  </head>
  <body>
    <main>
        <h1>Welcome to {{ vhost_domain }}</h1>  
    </main>
  </body>
</html>

roles/webserver/templates/site-vhost.conf.j2

<VirtualHost *:80>
    ServerName {{ vhost_domain }}
    Timeout {{ proxy_timeout }}
    KeepAliveTimeout {{ proxy_timeout }}
    MaxKeepAliveRequests 0

    ServerAlias www.{{ vhost_domain }}

    DocumentRoot /var/www/vhosts/{{ vhost_domain }}/public_html

    <Directory /var/www/vhosts/{{ vhost_domain }}/public_html>
        Options Indexes FollowSymLinks MultiViews
        AllowOverride All
    </Directory>



    CustomLog /var/log/apache2/{{ vhost_domain }}-access.log combined

    ErrorLog /var/log/apache2/{{ vhost_domain }}-error.log
    # Possible values include: debug, info, notice, warn, error, crit,

    # alert, emerg.

    LogLevel warn

</VirtualHost>

roles/webserver/vars/main.yaml

httpd_config_folder: /etc/apache2
vhost_domain: example.local
proxy_timeout: 60

roles/webserver/tasks/main.yaml

- name: Ensure apache is at the latest version
  ansible.builtin.apt:
    name: apache2
    state: latest
- name: Creates the html directory for site
  ansible.builtin.file:
    dest: '/var/www/vhosts/{{ vhost_domain }}/public_html'
    state: directory
- name: Create the site index.html
  ansible.builtin.template:
    src: ./templates/index.html.j2
    dest: '/var/www/vhosts/{{ vhost_domain }}/public_html/index.html'
- name: Config virtual host for site
  ansible.builtin.template:
    src: ./templates/site-vhost.conf.j2
    dest: '{{ httpd_config_folder }}/sites-available/{{ vhost_domain }}.conf'
  notify:
    - Restart apache
- name: Disables apache default vhost which conflicts with others
  ansible.builtin.command: "a2dissite 000-default.conf"
  args:
    removes: '{{ httpd_config_folder }}/sites-enabled/000-default.conf'
  notify:
    - Restart apache
- name: Enable VHost for Domain {{ vhost_domain }}  in apache
  ansible.builtin.command: "a2ensite {{ vhost_domain }}.conf"
  args:
    creates: '{{ httpd_config_folder }}/sites-enabled/{{ vhost_domain }}.conf'
  notify:
    - Restart apache
- name: Ensure Apache2 is Started
  ansible.builtin.service:
    name: apache2
    state: started

As you can see, we just refactor your previous site.yaml into separated chuncks. Ansible understands this role layout and put things in order for us.

To finalize lets check your databases.yaml, webservers.yaml, and site.yaml

databases.yaml

---
- hosts: databases
  become: true
  roles:
    - postgresql

webservers.yaml

---
- hosts: webservers
  become: true
  roles:
    - webserver

site.yaml

- import_playbook: webservers.yaml
- import_playbook: databases.yaml

The databases.yaml file targets the hosts databases and apply the role database, the webservers.yaml file do the same for webservers group. Finally site.yaml file glues the entire playbook together importing other playbooks.

We can then run this playbook on the local-hyperv environment with the command:

ansible-playbook -i inventories/local-hyperv site.yaml

If we need to change environments, we just change the inventories folder.

Final Thoughts

In this tutorial we cover the basics of Ansible so you can get started automating and documenting your infrastructure in sharable, replicable playbooks. One of core advantages of Ansible is it's lower starter requirements and easy to understand semantics.

Ansible Tower and Ansible Automation Platform are two products by Red

Hat that covers some limitations in Ansible core, by example its distributed authentication. Is the responsible of user to have root access to all machines through SSH, this could be a pain to maintain and rotate credentials in a big team. Another limitation of ansible core is if you want control who has access to different machines and environments, for the same reasons as the later, and also track which playbook has been applied and who applied it.

But Ansible core will serve you well if you do not have this specific requirements.

There is of course, some alternatives and complements to Ansible. I can mentioning a few with a brief summary of what it does:

• SaltStack - Main competitor of ansible, it has a different architecture using agent approach and a reactive event-driven infrastructure. The event drive means you can react to events on systems and proactively apply configurations. It also more performant than Ansible, it can run commands in thousands of systems in seconds. The agent approach means you need a master server, the advantage is that out of the box, Salt centralize the authentication of systems in the master node, so you don't have to mess up with distributed keys. Salt can also be used without the master-client using salt-ssh or salt-proxy. It has a lot of vocabulary to its components like: salt mine, salt minion, pillar, grains, wheel, etc. Which can be a little confuse at start. It's worth mention that it also has powerful tools in its ecosystem like the SaltStack Config that enables role based access control, multi master support, reporting, and others. More info in https://saltproject.io/
• Terraform - Terraform has a different objective than Ansible. Terraform is focused on provision of infrastructure, Ansible is focused on the post provision of it. That means terraform is good to automate cloud infra resource allocation and configuration. Terraform uses a custom language to define its resources. It is complementary to Ansible. More info https://www.terraform.io/
• Pulumi - A competitor for terraform, its main focus is also in provision of infrastructure, the main difference is that uses a couple of mainstream programming languages such as Javascript, Typescript, Python, Go, C#, Java and Yaml. It's complementary to Ansible. More info in https://www.pulumi.com/

All examples for this tutorial can be downloaded from https://github.com/giovannicandido/ansible-blog-post

References

https://docs.ansible.com/ansible-core/2.13/index.html

https://docs.ansible.com/ansible-core/2.13/getting_started/get_started_inventory.html

https://docs.ansible.com/ansible-core/2.13/user_guide/intro_adhoc.html#intro-adhoc

https://www.reddit.com/r/ansible/comments/92ds1w/accept_all_host_keys_one_time/

https://docs.ansible.com/ansible/2.8/user_guide/playbooks_best_practices.html