Grouper Developers Forum

[Chris Hyzer <>]

Oracle should have time in the DATE column.

To see which jobs are running, can you query for rows where there is no 
end_date, but yes start date?  Something like that?

Something like this?

select * from grouper_loader_log where ended_time is null and started_time > 
sysdate - 2

For loader jobs there should be insert/update/delete, for other jobs maybe not

Pending jobs are in the same table with no ended_time

Thanks,
Chris


-----Original Message-----
From: John Tran 
[mailto:]
 
Sent: Thursday, August 27, 2015 10:27 AM
To: Chris Hyzer
Cc: 

Subject: Re: [grouper-dev] Grouper Processing Speed

Chris, 

Thanks again for the reply.  I missed your email again, but I do believe I 
found the issue with my mail client now.  Apologies once again.  I’ve 
included some answers as well as followup questions below.


> On Aug 24, 2015, at 8:30 AM, Chris Hyzer 
> <>
>  wrote:
> 
>> In the grouper_loader_log, there is started_time and ended_time, but 
>> those are dates without timestamps.  
> 
> Are you in mysql (or something else non-oracle).  You could try changing 
> the column type from DATE to DATETIME and see if that helps (do that in 
> test first) :)

We are using Oracle.  I’ll talk to our admin and try that DATETIME switch!

> 
>> I see total_count which I assume 
>> is the total number of changes in that job.  
> 
> Total is the total number of members for a group or list of groups.  The 
> insert/update/delete shows how many have been changed, though there isn’t 
> something to say how many will be changed.

The insert, update, and delete fields are always zero.  A majority of the 
time total_count is at zero as well.  I assume it’s an issue with our 
configuration somewhere.

> 
>> However, I don’t see a 
>> field that would indicate how many have been processed and/or how many 
>> are left to process in that job. 
> 
> If you look at a job in process, it should update that record in the table 
> with progress.  You are looking at jobs which are status SUCCESS which 
> means done

The main reason for this inquiry is to see if we’re able to see how large the 
current and pending jobs are and to see how far along we are.  This will 
allow us to give our customers time estimates when there is a processing 
delay.  Are you aware of any available data to determine the number of 
records to be processed for a particular job?  It seems the answer is no as 
you stated above that there isn’t something to say how many will be changed.  
 

> 
>> If I’m correct to assume that grouper_change_log_entry_temp are the jobs 
>> in 
>> queue, is there a way to see how large these jobs are?
> 
> No, that is just the change log temp to change log job, copies records to a 
> different table and orders them and assigns ID's and maybe other things

Is there a table that contains the pending jobs?

> 
> Thanks,
> Chris
> 
> -----Original Message-----
> From: John Tran 
> [mailto:]
>  
> Sent: Monday, August 24, 2015 9:18 AM
> To: Chris Hyzer
> Cc: John Tran; 
> 
> Subject: Re: [grouper-dev] Grouper Processing Speed
> 
> Chris,
> 
> Apologies, I just came across your response this morning.  I’m not quite 
> sure how I missed it.    
> 
> In the grouper_loader_log, there is started_time and ended_time, but those 
> are dates without timestamps.  I see total_count which I assume is the 
> total number of changes in that job.  However, I don’t see a field that 
> would indicate how many have been processed and/or how many are left to 
> process in that job. 
> 
> If I’m correct to assume that grouper_change_log_entry_temp are the jobs in 
> queue, is there a way to see how large these jobs are?
> 
> Thanks,
> 
> John Tran
> IS Technical Services Senior
> Division of Information Technology
> University of Wisconsin - Madison
> 608.890.2971
> 
>> On Jul 30, 2015, at 10:51 AM, Chris Hyzer 
>> <>
>>  wrote:
>> 
>> Does the grouper_loader_log have what you need for those jobs?
>> 
>> -----Original Message-----
>> From: 
>> 
>>  
>> [mailto:]
>>  On Behalf Of 
>> 
>> Sent: Wednesday, July 29, 2015 4:37 PM
>> To: 
>> 
>> Subject: [grouper-dev] Grouper Processing Speed
>> 
>> Hello All,
>> 
>> We are in need of a solution that will give us pertinent processing
>> information during CHANGE_LOG_changeLogTempToChangeLog jobs, specifically 
>> ones
>> with a large amount of changes.  Currently, I can see when the job has
>> started, but haven't been able to figure out much more around processing.  
>>   
>> 
>> How can we see how many changes are in the current running job?  Is there a
>> way to view the progress?  
>> 
>> On a related note, it would be nice to see at what rate changes are
>> processing.  We'd like to get a baseline, then monitor whether processing 
>> is
>> slower than normal.
>> 
>> Thanks for your time,
>> 
>> John Tran
>> Department of Information Services
>> UW-Madison 
>