System replaces visual checks that are long and costly and do not perform a full scan.. Areas where gaps are found can be filled with plants when they are detected in time to make an intervention in the same crop season. In images with more than one hundred plants, the counting errors of the system were of about six units.. Experiments were carried out in maize crops and citrus orchards in the Southeast and Mid-West regions of Brazil. Work was a pioneer in using the convolutional neural network (CNN) method in this type of application. A collaborative network with professors and researchers from (both national and international) public and private institutions developed a pioneering solution in Brazil, which detects and counts plants while it identifies plantation rows in images obtained with drones. The task is performed thanks to a combination of advanced computer vision techniques and deep learning, capable of making decisions on its own. It reduces costs and uncertainties, facilitates sustainable crop management, and boosts agro 4.0. In experiments with maize and citrus crops in the Mid-West and Southeast regions of Brazil, the method has reached a high success rate in the monitoring of agricultural systems, in addition to showing versatility and allowing the reduction of the dependency on visual checks that are time-consuming, laborious, and biased. Another advantage in comparison with traditional methods is that the proposed solution allows a complete scanning of the plot or planted area. The accurate mapping of cultivation areas is an important precondition to help in field management and yield forecast in the area known as precision agriculture. It is because the crops are sensitive to the sowing patterns and have a limited capacity to compensate absent areas in a given row, which negatively impacts the yield per unit of soil area during the harvest time. Identifying plantation rows can help farmers correct problems that have happened during the cultivation of seedlings, an essential information in decision making. Thus, optical images with sensors loaded on unmanned aerial vehicles (UAVs) are a low-cost means commonly used to capture scenes and cover cultivated areas. Versatility and precision The study involved researchers from the Federal University of Mato Grosso do Sul (UFMS), University of Western São Paulo (Unoeste), Santa Catarina State University (Udesc), University of Waterloo, in Canada, and Embrapa Instrumentation. The proposal of the group was to develop a deep learning method based on a convolutional neural network (CNN) to simultaneously count and detect plants and plantation rows with images obtained by sensors boarded onto UAVs. Supported by the National Council for Scientific and Technological Development (CNPq) and the Coordination for the Improvement of Higher Education Personnel (Capes), the research is one of the results of the project on technologies with disruptive potential for automation and precision agriculture, led by the Embrapa researcher Lúcio André de Castro Jorge, a specialist in processing images captured by several types of drones. Art: Lúcio Jorge and Lucas Osco, translated by Mariana Medeiros The research The study was carried out with maize plants in early stages but with high density, in an experimental area at the School Farm of the Federal University of Mato Grosso do Sul, with an approximate area of 7,435 m². The research covered a total of 33,360 cornstalks in 224 plant rows. The method has reached a high performance for counting, missing approximately six plants per image, each one containing more than 100 plants, and a similar performance in the location and extraction of plantation rows. In citrus, the method was equally superior to other neural networks previously developed in other studies, missing between one and two trees per image. In maize fields, the areas with gaps can be filled with plants of the same crop in case they are detected in time to make an intervention in the same season. This condition occurs in different crops such as sugarcane, soybeans, tomatoes, among others with similar characteristics. Keeping an eye on this gap, researchers focused on a solution that could be replicated in other crops, not restricted only to maize and citrus fields. Honorable Mention and publications The research "A CNN approach to simultaneously count plants and detect plantation-rows from UAV imagery" received an Honorable Mention at the Mercosul Science and Technology Award - 2020, promoted by the Brazilian Ministry of Science, Technology and Innovation (MCTI) and the National Council for Scientific and Technological Development (CNPq). The award addressed the Artificial Intelligence topic, with six sub-themes. The Mercosul Award was released simultaneously in Argentina, Bolivia, Brazil, Chile, Colombia, Ecuador, Paraguay, Peru, and Uruguay. The study was published on the ISPRS Journal of Photogrammetry and Remote Sensing, by the International Society of Photogrammetry and Remote Sensing (ISPRS), in February. To access the article, this link and a free version of the paper can be accessed here. Another important contribution from the method is the detection of crops planted in high density or condensed with reduced spacing. The plants in the images of the experimental area were identified by photo-interpretation. The Unoeste professor Lucas Prado Osco, supervised by the researcher José Marcato Junior during his post-doctorate at UFMS, explains that the data were inserted in the neural network as an example for learning. "This way the method learns from such examples. What happens is that plants are very close to each other and it can be a problematic factor for conventional deep learning methods. This method uses an approach in which the probability of each pixel being a plant is real and, through smart refining, it can define the central pixel and detect the position of the plant in the image," details Osco, who is a scholarship holder at Embrapa Instrumentation in the project on disruptive technologies. According to Castro Jorge, none of the studies implemented a detection of plantation rows in their methods with convolutional neural networks, another differential of the current approach. "Although many deep networks of object detection can be used to detect plants and plantation rows, they require several steps of image processing with highly costly conventional techniques and modifications to run both tasks together," the researcher compares. The proposed approach uses a two-branch architecture, a model that allows information exchange between the network branches. "Thus, the detection of lines by the network benefits from plant detection learning, and vice-versa, once it understands that there will not be plants outside the lines and a line cannot be formed without the existence of plants. It also contributes to reducing the detection of weeds, even though future studies are still necessary to assess this condition more clearly," Castro Jorge reports. The researcher remembers that the evolutions in remote sensing technologies and computer vision methods suffered a disruptive advance with convolutional networks and are significantly improving the mapping of agricultural systems. "This integration is benefiting precision agriculture in many applications, such as environment control, phenological characterization, nutritional assessment, yield prediction, temporal analysis, crop management, among other benefits," he assesses. In face of the estimate by the Food and Agriculture Organization of the United Nations (FAO/UN) that, to meet the demand for food in 2050, agricultural production will need to increase more than 60%, with a participation of 41% of Brazil alone, it is expected that farmers increase their yield in the field. "However, this increase must come from technological advances and optimization instead of expansion of production areas . A precise estimate of plants in cultivation fields is important to predict the yield amount while monitoring their growth," the professor of the Faculty of Engineering, Architecture and Urbanism and Geography of UFMS José Marcato Junior says. For the professor, the detection of plants and lines of plants consists in an important measure in the assessment of agricultural fields because the number of plants helps farmers and rural technicians in estimating the yield at the end of the crop cycle. "This type of assessment, when performed in early stages of planting, is important for fast decision making. For maize and other crops, the decision window is brief and a fast detection can help mitigate or prevent production problems. Such practices should improve the applications of precision agriculture, resulting in the sustainable management of the agricultural system," he adds. Low cost as a differential A preliminary version of the method was applied for the first time to count citrus trees and it obtained a precision of approximately 97% of success. Both in citrus and maize, the group used images of a cultivated field, captured by cameras with RGB sensors loaded in drones to compose the data set. The RGB system - a system of additive colors in which red, green, and blue are combined - is a low cost solution and because of that, it is installed in most drones, is easily replicable, and has high availability in the market. "The tendency to use RGB sensors allowed important results with reduced costs when compared to the use of special sensors in other light spectrum ranges. Thus, the method is a low cost and viable alternative to be applied to any crop. A great differential is still in the possibility of loading directly a smart system that allows detection in real-time from trained networks onto a UAV ," Castro Jorge evaluates. One of the main challenges involved the detection of plants in image borders, when most of them are obstructed. "The complication happens because of high occlusion regions, where one plant overlaps the other. In addition, another difficulty, this case with line detection, is related to the spacing between plants. There are lines in which, due to losses during planting, the distance between one plant and the other is large. It makes it difficult for network learning because it may not comprehend that one plant that is very distant from others can still belong to the same line," the scientist tells. "Notwithstanding, even in such few cases, we have observed that the published method is capable of correctly predicting the position of most plants and lines," Lucas Osco affirms. Future perspectives Scientists believe that future research and applications can benefit from the method developed to help deep neural networks in the simultaneous counting of plants and detection of planting rows in other types of crops. "We are implementing new features to the method in order to overcome different challenges related to planting standards. Also, we are confident with the current stage because it allows an improvement in decision making tasks at the same time that it contributes to a more sustainable management of agricultural systems," the UFMS professor José Marcato Junior concludes. Deep learning In simple terms, artificial neural networks are computer algorithms used with the intention of simulating the learning process of a biological brain to extract and recognize information and patterns. Such networks have been gaining more and more space in data analysis, especially in recent years. "Deep learning is a type of machine learning technique that uses complex and deep artificial neural networks to learn a pattern and extract information from it. This technique has been used in several applications in the last years and it has gained popularity in tasks related to remote sensing and precision agriculture. However, it requires a substantial amount of labeled examples to learn, but once it has learned it can apply its knowledge in different scenarios and conditions, as it is a highly generalizing method," Wesley Nunes Gonçalves adds, professor at the Faculty of Computing of UFMS and responsible for the development of the method applied.