Introduction
AK: Konnichiwa, welcome to the AI automation dojo. Today we are dissecting the delicate art of letting artificial intelligence judge human potential, specifically looking at a massive UiPath automation project we delivered for one of our clients. We’re joined by Aleksandra KwiecieĊ to break down how we built a cost-efficient AI agent that evaluates applicant resumes against actual job requirements, safely locked away from the rest of the corporate databases so it can’t accidentally email the CEO a rejection letter. I’m your host Andrzej Kinastowski, one of the founders of Office Samurai, where we firmly believe that the only thing harder than finding the perfect candidate is explaining to an algorithm why rockstar developer isn’t a literal music profession. So whether you’re a talent acquisition specialist drowning in a sea of generic cover letters generated by ChatGPT or a developer trying to figure out how to stop an AI from brutally rejecting every junior applicant who applies, you’re in the right place. Now grab your favorite katana or just the nearest stack of unsorted candidate portfolios and let’s get to it.
Today we have Ola KwiecieĊ with us. Ola joined Office Samurai more than six years ago as a junior developer and in that time she has become a senior developer. She is an automation lead and she is working with AI a lot, especially UiPath agents and UiPath Maestro. She was a part of a few projects where AI or Gen AI specifically was a big technological part. One of them actually we had an episode about some time ago, it was the one with reading stacks of PDF documents. Actually the customer has just presented this project on a UiPath Fusion event in Warsaw. They told the story about how they approached eight different vendors about solving this problem. Six of them said it’s not possible, two of them said yes it’s doable, they did POCs with both of them and Office Samurai was the one that delivered the project in the end, which I thought was really nice. So Ola has a lot of experience with AI and Agentic and this is what we would like to talk about today. Ola, welcome to the dojo.
OK: Hi, thank you for having me.
How agents fit into automation efforts
AK: Listen, you have been doing those AI projects for quite some time. You have also been a part of the Fast Track last year when we were learning to use UiPath’s implementation of agents. I would like to start before we get to the particular project we want to talk about, I wanted to ask you how do you feel those agents fit into all of the automation efforts that we do or that corporations do actually?
OK: AI technologies can extend the automation a lot. For the time that I get some requests from our clients regarding using agents or other AI technologies, I’m often really surprised about the projects, actually processes they are asking for. Because previously I haven’t seen processes like this in my life. For example, processes where you have to check something on the map. This is absolutely not possible for the RPA automation. For example, checking something on documents, so for example is there a signature or I had also a request: is the signature legit? So whether it fits the original signature of somebody’s for example. I guess because we haven’t delivered this project yet and haven’t analyzed this project yet actually, but those requests are from all of the fields that you can imagine actually. Also the HR departments and those processes that are really close to humans, they were always really hard for us to automate somehow and right now with AI technologies, I mean all of them, IDP solutions, sometimes probably NLP but also Agentic, made them possible to be analyzed and also maybe automated at the end.
AK: I have the same feeling that processes that customers would come to us with a few years ago and we would tell them it’s not possible, nowadays we’re not so sure anymore. We usually say something we would need to check, right?
OK: We would need to check. Yes, exactly.
The issue of high-volume recruitment
AK: You started talking about HR and this is what I wanted to talk about today. You have been a part of the project where we have been automating administration around HR recruitment and this is something that is relevant for a lot of big, basically any big corporation, because there’s a lot of recruitment. The bigger the company, the more recruitment you have and you get a lot of CVs coming in and you never have enough people in recruitment to process everything. So could you tell us what was the issue that the customer came to us with with this particular project?
OK: Actually, exactly, a lot of CVs and not too many people to process them. If I’m not mistaken, there are only three people in our customer’s HR department and they have a few recruitment processes every month with volumes like 1,300 CVs for each. So the volume was huge and their problem was that actually those three people had really a lot of problems with processing all of them. So they knew that probably there are really good candidates in this batch of CVs but sometimes they don’t even have time to read those CVs to open the application. So those candidates can be lost in this huge number of CVs. So they wanted to automate the process to help the people and to let them focus on the best candidates.
AK: But we’re not talking the agent hiring the people, right? The agent would not go to the recruitment meeting and make a decision, right? We are talking the things that happened before that.
OK: When we are thinking about the recruitment process, I guess today’s technologies probably could automate most of the steps but is it okay for the candidates? I’m not sure. So in our process, we still have a lot of human effort. So the only thing that automation does, it checks all of the documents of the candidate. So it checks not only CVs but also motivational letters, anything that the candidate could put into the application and the agent is checking whether the candidate meets the requirements. So we have the list of requirements and that’s the only thing that the agent does, the requirement can be met or not. After that we have an RPA solution that calculates the grade for the candidate but the decision at the end is still human.
How the agent checks requirements
AK: Okay, as it should be. But so if I understand correctly, we get a bunch of CVs, we’ve got some sort of a job description with certain requirements and the agent needs to compare the content of the CV and other materials that the person sent to those requirements. And does it grade it somehow, how does this work?
OK: No, the agent does not grade anything. It only decides, so it says met or not met, yes exactly. And there are a lot of different requirements. Some of them are really easy to check for the agent. Really easy. For example languages. So we have a requirement that the candidate should speak in English at least at B1 level for example. So this is something that the candidate nearly always puts into the CV. But if they put it in a different way, like if they say intermediate or advanced or something, will the agent still get it?
OK: Yes, the agent understands the content. So it’s not a problem. But we had actually one problem with languages because during the testing phase I guess we saw that we need to check also Polish. And the problem was that sometimes Polish candidates have their CVs in English. They have Polish names, so we guess that they are Polish right, but they will not write Polish native for example. So they consider it to be completely obvious that they speak Polish and they don’t even mention it on the CVs.
AK: If their name is Jan Kowalski we can assume that this person is Polish but there’s no information in the CV that he knows Polish. Okay, so this was a hard one for the agent. But back to the subject. So some of the requirements are easy to check but some of them are related to soft skills for example.
OK: So we have requirements like communication in the context of business relations for example. We don’t mean communication like if someone is nice to teammates or something like this but more something about the negotiations with the client or getting new clients. So this is something harder to check. Sometimes people are writing in their CVs that they had to negotiate with the client in the previous job, then it’s easy for the agent. But sometimes it’s just a general feeling.
Educated guesses and non-deterministic results
AK: But it would also be for a human, right? So a human would also have to make like an educated guess from reading the CV.
OK: Yes. And the agent actually is also making an educated guess because on the same input, the same CV, it can make different decisions. So one time we are processing one CV, the agent can decide that yes, this candidate meets this requirement, and another time it will tell us: oh no no no no I’m not so sure, so I would say it’s not met. And this is something normal. I mean those are non-deterministic technologies so you do get a certain level of chaos. Even if you drop the temperature to zero you can still get different answers right?
OK: Yes, we have temperature set on zero and still different answers, and mainly for those soft skills related requirements, then this is normal.
AK: Okay. But I think it’s worth noting that the same human would also judge the same CV differently on different days or before lunch and after lunch.
OK: Yes, exactly.
AK: So in that way those algorithms are kind of made in our image. Okay. And when it comes to how the CV is presented, did you get problems with like some people will make a fancy CV with little stars instead of levels and things like that? Is this something that an agent would be able to understand, did you test this?
OK: I don’t think we had an example like this. But we didn’t also have any feedback from the client that there is something wrong with the language because those are requirements that really lowers the grade.
The grading process and manager interviews
AK: In a typical job advert how many of those requirements is the agent assessing right now?
OK: I guess about 15 different requirements but we also divided those requirements into base requirements and additional ones. The base requirements are those that are mentioned in the job offer and the additional ones are related to something that our client thinks will be beneficial. So there are requirements like for example sports. They really like people who do sports because they are competitive, they have a team spirit also. So this is an additional requirement that obviously the candidate does not have to meet but it gives them some additional yes, right. And it says something about their character, which was really important for this particular job.
AK: Okay. So I understand the automation takes in the CVs, the agent for each CV grades how well it fits the profile that the company needs right, and what happens then?
OK: Then we are putting this grade into the HR system and also we are sending the best candidates, I mean with four or five stars, to the managers that can meet with them for the interview. So at the end of the process there is always a human because the people that are getting one, two, three stars, they are sent to the HR department and their CVs are still checked by the HR department. But for the candidates that have higher grades they are sent directly to the managers and they can decide if they want to meet with this person or not.
AK: And those that get one, two or three, once they screen them do they come back to you saying: “Hey what happened here I like this person.”
OK: Yeah. They are always coming to us with questions like this if there is a huge gap between the agent’s grade and the human grade. So they know that agent can make different decisions each time, but they also know that if the grade is entirely different, so for example our agent will grade someone a one and they think that the person should be a three or four, and then they are coming to us. We had a few situations like this with a bigger batch of those CVs. So in that case we are always checking if there is something in common between those CVs.
Learning from feedback: the student example
AK: Okay. So do you have any examples of what the agent was getting wrong or not in line with the people’s expectations?
OK: Yes. For example students. In the past it didn’t like students because the students do not have much experience. So they were not meeting a lot of requirements but for example we could see that they made a lot of additional projects for example or they are a part of some kind of student associations and so on. So from that you know that they can be motivated, highly motivated right, interested in the job and that they are willing to take. So they are really interested in the job they are applying for.
AK: Okay. But so what do you do when you get this kind of information? So you learn from the people checking the results, you learn from them that: okay it grades students too low, we wanted to add them some points for instance for those extracurricular activities. What do you do then?
OK: The first step is the analysis. So sometimes the ladies from HR just coming to us and saying: “hey we are not grading students for example enough”, but sometimes this is something that we are analyzing and asking them: “okay I think that we could grade students a bit low, maybe we should change that.” And this is always a discussion. So we have an idea, we have to discuss it and make a decision if we want to add something to the agent because it is not a simple task. So like in the RPA we have a bug, we have to find a place and we have to fix it, right? And in that case we can have some ideas. We have to be also careful while making any changes in those list of requirements and after that you also have to test it on those CVs but also on other CVs if we didn’t mess something up.
AK: Yeah, that’s the thing with working with agents. You get feedback, you change the prompt or other settings, but while fixing one problem you might have created another right? So I’m really glad you touched on testing because this we have learned that with GenAI projects testing is a huge part of the whole project, way bigger than with RPA or even with like classical AI tools. How do we do this in this project, how was it done?
OK: At first I got a bunch of CVs from the ladies from HR and I knew what grades they have. So I had a evaluation set let’s say. And that I know the answer from the human user. So this was something that I was always coming back to. If I had any new requirement I had to test it on this evaluation set and check if okay, so for the CV we fixed something but maybe five other CVs have different scores than it should.
AK: Okay. So automated testing that you can run at any time if you change something like you change a model, you change a prompt, and then you see whether the results are better or worse right?
OK: Yes. Okay. But right now we are really careful about those requirements and changing them because we actually processed about three or more even recruitment processes on production. So a few thousand CVs already I guess. And we know that more than less CVs are processed correctly. So right now I’m really cautious if there are any ideas for the change, but probably in the future the client would want to have other profiles for their employees.
Business reaction to AI automation
AK: So this has gone into production. It’s working. It’s saving the people’s time. What’s the reaction of the business? How was it like? Because you know people fear AI taking their jobs right, we read all those articles about how you know 20% unemployment and all of this stuff. How were those discussions with the business and how happy or unhappy are they with the solution?
OK: Well this particular client is definitely not scared of automations and huge ideas. They even got this innovation prize on the conference that you mentioned at the very beginning. So they were really excited about it. They were also really prepared so they had I think at least a few discussions inside their organization. And they knew that if this agent will be good enough then there is a lot of time to be saved because right now, the processing time, so this gap between the application comes to the system and the application comes to the manager that can actually make a decision, it’s about two days I guess, and before it was in one week or two weeks. So this difference is really huge.
AK: Okay. So it’s not just about saving the time of the people in HR but it is also about being able to hire people faster.
OK: Yes. There were a lot of issues that some candidates that were really good for this job just took another one in the meantime.
AK: Okay, right. Yeah, that makes a lot of sense. Being in a recruitment process that drags is a really frustrating thing for people to endure. And in here we are looking for highly motivated people. So you can imagine they are not waiting. I would say so. And I think if the company is looking for motivated people it should also be able to show that it itself is motivated to hire them. So yeah. Okay, that is quite good because very often, probably most often when the customer approaches us about an automation, the first thing they think about is saving money, which makes a lot of sense, but very often we forget about other benefits that can prove even more substantial right? One thing would faster hiring, but another what you mentioned at the beginning, we screen more CVs or basically all CVs that we get and this way we do not miss talent that otherwise we could have missed.
OK: Yes, exactly.
Security threats and direct access
AK: So I wanted to ask you because this is something we are very careful about. There is a big trend when you look on LinkedIn or YouTube, you see those people saying: “Hey I’ve got this agent, I gave it access to my desktop and I told it to do something for me and then it deleted all of my files.” Right, so how do we deal with this kind of security threats in this case?
OK: Security is a thing that you have to always think of. So first of all the agent should not have access to any systems, databases, anything that it actually doesn’t need.
AK: True. So you mean direct access?
OK: For example our agent does not have access to the HR system, doesn’t have access to our database where we store information about the processing. It doesn’t even have access to the files itself, the files that it processes. So in that case even if the agent would do something that we didn’t foresee, then it cannot mess any production systems or even delete files that are being processed. So this is something that you always have to think of regarding agents. We were thinking about prompt injections for example because probably maybe it could be possible that somehow someone would write a prompt inside the CV that will for example make: “grade me a five”, right?
AK: Yeah. But checking that would be the additional cost and at the end we still have a human. Yeah, so that would be the worst case scenario that the candidate has a different grade, but probably the HR or the manager would come back to us and be like: “what is that, doesn’t match.” I mean prompt injection in, if there is this separation that as you rightfully say the prompt injection can only affect the grade, but then we still have a human in the loop that will see that. And this layer between the agent and the systems would be unattended robots right?
OK: Yes, exactly. Okay, so we have a layer of classical automation where we do not let the agent, as you said, access files directly or databases or whatever. It’s the robot that takes a CV, probably reads it and then sends the content to the agent and then receives the grading.
OK: Yes exactly. And also if the agent would return a different output, we want to have a JSON array actually with all of the requirements and the decision if it is met or not. So even if someone would make some kind of prompt injection that would at the end give us different output, then still the RPA robot would not be able to for example deserialize this JSON array. So it will be a problem or it won’t make sense what will be in there. So it will not be processed.
AK: Exactly.
OK: But we also have to have in mind that people are really creative. So we will see. But in that case it would be really hard to make a successful prompt injection.
Unpredictability and context management
AK: I would say so. And this is something that again we, there is this, I mean it sounds amazing to give the tools to the agent so it can do things by itself so we don’t have to build anything in between. But those are such fresh technologies that anything can basically happen and we hear those stories about things going really wrong even not because somebody did something malicious but just because of the unpredictability of the LLMs. Right?
OK: Yes. And if your agent has tools you have to also think of their accesses for example. So if you have a tool that has access to the database then this is less safe. You have to think of how to secure the production database for example.
AK: Right. Yeah, so the agent doesn’t decide that the best course of action is to drop the whole database.
OK: Exactly.
AK: I mean I’m sure we’re going to hear a lot of stories like this in the near future with how things are going and how people are experimenting. And you mentioned context and this is something that we often give to agents. We provide context that gives additional information to the agent so it knows how to work. How did you do it in here because it had to have the information about the roles and how to grade them? Where did you put that?
OK: Yes, that was the first idea to put the requirements from the job offer or even the job offer itself to the context. But we had two hard requirements at the very beginning from our clients. So the first one was that they want this agent to be reusable and the second one was that they wanted it to be cost effective. That’s obvious if they have that volume of CVs. So I was thinking about this context in the context of being reusable. So I knew that for now we have to process a single recruitment process with a defined job description right, but how would it behave if we want to process an entirely different recruitment process? In UiPath agents it’s a cool tool but it’s also low code, so you have less control over the context that the agent uses at this moment. So I decided not to use the context in the UiPath agents tool but giving the agent this context as an input variable. So all of the requirements are being put into the input argument for the agent and with that solution, if we have another recruitment process, then you only need to change the input. And also the agent does not have too much context because if we would be adding the recruitment processes then the context probably would be huge.
AK: I mean and the bigger the context, zwĊaszcza if it’s similar data and in this case it would be stacks of similar data, then probably the quality of it would be lower the more context we add to it. So I understand you do not put the context in the agent itself, there is a prompt that is as generic as possible and then in the input we put both the contents of the CV and the requirements for this particular job. So you can basically use exactly the same agent for a few different jobs.OK: Yes. So this agent in the UiPath Maestro tool seems like it doesn’t have context and tools but actually the context is the input argument and the tools are in the unattended RPA robot.
Developing for agents vs. classical RPA
AK: Okay, that makes sense. I mean for sure from the reusability and from the cost perspective that makes a lot of sense for this project. So kind of taking a step back because you have been developing RPA bots for more than six years now and it was a different world when we were starting right, the technology was not as polished as it is today. But looking at your experience with developing automations with pure RPA and creating agents that are still used within the RPA platform but it’s like a completely different way of doing things. How would you compare this? Is it easier or harder, we know it allows you to do more, but from the perspective of a developer how what differences or similarities you see when it comes to classical automation and agents?
OK: I think that with agents and AI technologies in general there are more unknowns. Like we mentioned before, if a customer is asking us for analyzing a process for AI, we often say: “okay it seems possible but we have to check it.” And sometimes even after checking it we are still not really sure what will be the final efficiency of this AI solution. In RPA, of course even if we have the senior developer with a lot of experience, sometimes the developer doesn’t know something and have to check it for example, but in AI I think there is just much more question marks right?
AK: Yeah, I mean with pure RPA we see another process that takes data from one system and push it into another and we know we’ve done that a lot of times. So it’s clear that we can do it and we more or less know how much work it’s going to be right. And with AI it is as you said more trial and error still and the end result is unknown from the perspective of we know it can be done but we don’t know if it’s going to be 60% accuracy or 90% or 95 right?
OK: Yes. Also with RPA we would like to exclude the human as much as possible right. We don’t want the human to help the RPA robot. In AI we definitely still want the human in our process. So you have to design it that way that when the agent makes any decision, somewhere in between there should be a human.
Where to start with AI automation
AK: Okay. And from the perspective of how the projects look like, like the structure of them, do you see that they differ from classical RPA projects? Sometimes I was thinking about documentation of a project like this HR agent because sometimes there are processes that can be mapped, the processes that are being automated with AI because somewhere on the map on the flow of the process there is a decision that has to be taken by the AI. But in case of this HR agent I was thinking okay, how to make a map with the decision process of this agent?
AK: Yeah, because it’s like one box, it’s an agent that gets a prompt and then returns a result but there’s a lot of logic within it.
OK: Yes. So I think this is something that would differ in every process. But also regarding the architecture: where is the human, how do we communicate with the human, where should we put some guard rails in the process so they are designed more to be closer to the human I would say.
AK: What would you say to companies because a lot of companies are now wanting to get into Agentic right, it’s all the rage right now. And but they don’t know where to start right and they don’t know what would be a good candidate for an automation. The CFO came and said: “I want three AI implemented and now you need to figure out something.” What would you say to them where they should be looking in terms of what is automatable with AI and maybe if there are any areas where they shouldn’t be looking or maybe not at first?
OK: I would probably start with the areas that you already have the automation. And for example there are often places where one automation has to pass the information into a human, human does something and it is then put into the another automation. That would be the exact place where AI could help. Also departments like HR in our example, those are departments that are not automated a lot from obvious reasons. I would also think of processes that have high volumes. So the higher volume the better for the AI because we need a lot of data to test the AI, to design how it should think. Right, so that would be the places that I would start with.
Maintenance and ongoing supervision
AK: And when they do find something is it like all the big consultings are promising us that you can just give it to the agent and forget it or what’s your experience?
OK: Definitely not and you definitely shouldn’t. The AI has to be checked by the human always. Always. It’s not like you have your process automated and you can forget about it right. The AI can change the decision as we mentioned. AI can hallucinate. There can be like really a lot of problems with AI that we want to minimize obviously. But those are decisions that are not based on rules. So sometimes there can be a decision that is taken by the AI that we are not comfortable with. That is the place when the human should stop the process or change the decision. Yeah, so definitely there always has to be a human in those processes and you have to think about it while you are designing the solution.
AK: So the agents can automate a lot of things that we do but it’s almost never 100% of cases and it’s almost never good enough just to let it run the process without supervision. And also I would presume that in time it may require some maintenance or adjustment because I don’t know people start preparing CVs in different ways or we open different kinds of positions that maybe the prompt needs some fine tuning. So it’s not a fire and forget. It is something that like all automations will need some work in the future.
OK: Yes exactly. We can also try to extend the automation, add another agent or add another tool for the agent. So definitely we still have the maintenance that we have in RPA automations. But we also have this maintenance day by day if there are decisions where the agent should behave differently for example. And we have to also remember that agents and LLM models in general are consuming tokens. So this is something that is also happening on a daily basis.
AK: And when they do find something is it like all the big consultings are promising us that you can just give it to the agent and forget it or what’s your experience?
OK: Definitely not and you definitely shouldn’t. The AI has to be checked by the human always. Always. It’s not like you have your process automated and you can forget about it right. The AI can change the decision as we mentioned. AI can hallucinate. There can be like really a lot of problems with AI that we want to minimize obviously. But those are decisions that are not based on rules. So sometimes there can be a decision that is taken by the AI that we are not comfortable with. That is the place when the human should stop the process or change the decision. Yeah, so definitely there always has to be a human in those processes and you have to think about it while you are designing the solution.
AK: So the agents can automate a lot of things that we do but it’s almost never 100% of cases and it’s almost never good enough just to let it run the process without supervision. And also I would presume that in time it may require some maintenance or adjustment because I don’t know people start preparing CVs in different ways or we open different kinds of positions that maybe the prompt needs some fine tuning. So it’s not a fire and forget. It is something that like all automations will need some work in the future.
OK: Yes exactly. We can also try to extend the automation, add another agent or add another tool for the agent. So definitely we still have the maintenance that we have in RPA automations. But we also have this maintenance day by day if there are decisions where the agent should behave differently for example. And we have to also remember that agents and LLM models in general are consuming tokens. So this is something that is also happening on a daily basis.
Conclusion and final thoughts
AK: Yeah, they are not free. Okay. Ola my last question, with you having experience in different kinds of automation what would you say to developers? Is it fun to go into GenAI, is it frustrating, is it harder work, what’s your feeling about this?
OK: I think that it depends on the developer. For me this is extremely interesting. I like this moment when I don’t know what to say, when I have to check if it will work. But I am also aware that some developers prefer classical tools. And it’s also okay I think. Those technical skills are still important even if the LLM model can write the code for you. So it depend on a human at the end I would say. It can be harder but it also can be more exciting for me personally.
AK: Yeah, so it may depend on the person’s need for adventure versus…
OK: This is a kind of adventure, yes.
AK: Okay, that sounds good. Ola thank you so much for being here, for talking to us about the project. Looking forward to future projects that you’re going to be doing. Maybe in some time we can have you back to talk about another interesting implementation.
OK: Well we have two or three agents in the timeline right now, so probably I will have some new stories.
AK: Okay sounds good, thank you so much.
OK: Thank you.
AK: And that’s a wrap. The resumes have been passed, the algorithms have been thoroughly checked for bias against college students and the recruitment team is finally breathing a sigh of relief. Thank you for listening. We know your time is valuable, certainly more valuable than spending your afternoon trying to figure out if someone’s claim of being proficient in Microsoft Word means they know how to write the macro or if they just know how to use bold text. Huge thanks to Aleksandra for proving that humans and algorithms can peacefully coexist in HR and to our producer Anna Cubal, who continuously filters out my worst jokes before they reach the final cut. As always recorded in the pristine securely airgapped environment of Wodzu Beats Studio. We have created this episode in cooperation with UiPath, our first choice in AI automation. If this episode helped you see the light at the end of the hiring tunnel hit subscribe, leave a five star review and check out the Office Samurai AI automation playbook from the description. Until next time, may your applicant pipelines be full and your AI hallucinations contained. Mata ne!
