Grouper Developers Forum

[John Tran <>]

Chris,

Thanks for all your time.  I’ll do some more digging into the date and a 
couple other things that don’t seem to be working correctly on our end.  
You’ve been of great help.  

John Tran
IS Technical Services Senior
Division of Information Technology
University of Wisconsin - Madison
608.890.2971

> On Aug 27, 2015, at 11:29 AM, Chris Hyzer 
> <>
>  wrote:
> 
> Oracle should have time in the DATE column.
> 
> To see which jobs are running, can you query for rows where there is no 
> end_date, but yes start date?  Something like that?
> 
> Something like this?
> 
> select * from grouper_loader_log where ended_time is null and started_time 
> > sysdate - 2
> 
> For loader jobs there should be insert/update/delete, for other jobs maybe 
> not
> 
> Pending jobs are in the same table with no ended_time
> 
> Thanks,
> Chris
> 
> 
> -----Original Message-----
> From: John Tran 
> [mailto:]
>  
> Sent: Thursday, August 27, 2015 10:27 AM
> To: Chris Hyzer
> Cc: 
> 
> Subject: Re: [grouper-dev] Grouper Processing Speed
> 
> Chris, 
> 
> Thanks again for the reply.  I missed your email again, but I do believe I 
> found the issue with my mail client now.  Apologies once again.  I’ve 
> included some answers as well as followup questions below.
> 
> 
>> On Aug 24, 2015, at 8:30 AM, Chris Hyzer 
>> <>
>>  wrote:
>> 
>>> In the grouper_loader_log, there is started_time and ended_time, but 
>>> those are dates without timestamps.  
>> 
>> Are you in mysql (or something else non-oracle).  You could try changing 
>> the column type from DATE to DATETIME and see if that helps (do that in 
>> test first) :)
> 
> We are using Oracle.  I’ll talk to our admin and try that DATETIME switch!
> 
>> 
>>> I see total_count which I assume 
>>> is the total number of changes in that job.  
>> 
>> Total is the total number of members for a group or list of groups.  The 
>> insert/update/delete shows how many have been changed, though there isn’t 
>> something to say how many will be changed.
> 
> The insert, update, and delete fields are always zero.  A majority of the 
> time total_count is at zero as well.  I assume it’s an issue with our 
> configuration somewhere.
> 
>> 
>>> However, I don’t see a 
>>> field that would indicate how many have been processed and/or how many 
>>> are left to process in that job. 
>> 
>> If you look at a job in process, it should update that record in the table 
>> with progress.  You are looking at jobs which are status SUCCESS which 
>> means done
> 
> The main reason for this inquiry is to see if we’re able to see how large 
> the current and pending jobs are and to see how far along we are.  This 
> will allow us to give our customers time estimates when there is a 
> processing delay.  Are you aware of any available data to determine the 
> number of records to be processed for a particular job?  It seems the 
> answer is no as you stated above that there isn’t something to say how many 
> will be changed.   
> 
>> 
>>> If I’m correct to assume that grouper_change_log_entry_temp are the jobs 
>>> in 
>>> queue, is there a way to see how large these jobs are?
>> 
>> No, that is just the change log temp to change log job, copies records to 
>> a different table and orders them and assigns ID's and maybe other things
> 
> Is there a table that contains the pending jobs?
> 
>> 
>> Thanks,
>> Chris
>> 
>> -----Original Message-----
>> From: John Tran 
>> [mailto:]
>>  
>> Sent: Monday, August 24, 2015 9:18 AM
>> To: Chris Hyzer
>> Cc: John Tran; 
>> 
>> Subject: Re: [grouper-dev] Grouper Processing Speed
>> 
>> Chris,
>> 
>> Apologies, I just came across your response this morning.  I’m not quite 
>> sure how I missed it.    
>> 
>> In the grouper_loader_log, there is started_time and ended_time, but those 
>> are dates without timestamps.  I see total_count which I assume is the 
>> total number of changes in that job.  However, I don’t see a field that 
>> would indicate how many have been processed and/or how many are left to 
>> process in that job. 
>> 
>> If I’m correct to assume that grouper_change_log_entry_temp are the jobs 
>> in queue, is there a way to see how large these jobs are?
>> 
>> Thanks,
>> 
>> John Tran
>> IS Technical Services Senior
>> Division of Information Technology
>> University of Wisconsin - Madison
>> 608.890.2971
>> 
>>> On Jul 30, 2015, at 10:51 AM, Chris Hyzer 
>>> <>
>>>  wrote:
>>> 
>>> Does the grouper_loader_log have what you need for those jobs?
>>> 
>>> -----Original Message-----
>>> From: 
>>> 
>>>  
>>> [mailto:]
>>>  On Behalf Of 
>>> 
>>> Sent: Wednesday, July 29, 2015 4:37 PM
>>> To: 
>>> 
>>> Subject: [grouper-dev] Grouper Processing Speed
>>> 
>>> Hello All,
>>> 
>>> We are in need of a solution that will give us pertinent processing
>>> information during CHANGE_LOG_changeLogTempToChangeLog jobs, specifically 
>>> ones
>>> with a large amount of changes.  Currently, I can see when the job has
>>> started, but haven't been able to figure out much more around processing. 
>>>    
>>> 
>>> How can we see how many changes are in the current running job?  Is there 
>>> a
>>> way to view the progress?  
>>> 
>>> On a related note, it would be nice to see at what rate changes are
>>> processing.  We'd like to get a baseline, then monitor whether processing 
>>> is
>>> slower than normal.
>>> 
>>> Thanks for your time,
>>> 
>>> John Tran
>>> Department of Information Services
>>> UW-Madison 
>> 
>